User Patrick raised the following excellent questions to which I want to respond publicly, for everyone to benefit:
1. What do you mean by too many heavy connections hourly?
Hourly connections that transfer large data volumes (in the range of several 100.000 thousand cells) cause significant pressure on the update servers. The pressure is being further increased if those hourly connections are combined with features such as consolidate, transfer of formatting our query.
At this stage, it is impossible for us to determine exact limitations and specifications. We are constantly improving and optimising our server infrastructure, to address timeouts and increase the reliability of those connections.
Generally, we recommend to think thoroughly which connections are needed on an hourly basis. Those that are not, please consider moving them to daily or manual updates.
Lastly, every user has a set Google API quota that he can use to transfer data via Sheetgo. If this limit is reached–for instance by a large number of hourly updates–Google is preventing the update for that specific user.
2. What qualifies as "heavy"?
As stated above, heavy comprises among others:
- hourly connections
- large consolidations
- complex queries or filters
- complex source or destination sheets (many formulas, many sheet tabs, many cells)
3. Is it a Google-related limit or is it because we’re using too much of Sheetgo’s capacity?
It’s a mix of both. Sometimes, it’s the Sheetgo server timing out, sometimes the user may be exceeding the Google Sheets API quota. The latter we cannot influence and Google urged us to please not misuse Google Sheets as something they are designed for: a place for large data volumes. Please be considerate.
4. Is there a hard limit somewhere or does it vary according to the moment?
There are some hard limits. For instance, we are limiting Excel or CSV files to 10MB at the moment. But this may be increased moving forward. We are still experimenting with those hard limits.
Additionally, it varies according to the moment. At some times, many users have scheduled large unpredictive connections (maybe either source or destination has changed and Sheetgo doesn’t know). In such moments, the servers can simply collapse due to too high pressure.
5. Does it have to do with the moment they’re running, meanining that if we could space their running time, we might be able to solve it?
This definitely helps, yes! Try to not run all your connections at the same time. Instead, use smart schedules. Think twice when you need the data to be available. The fewer you update and the smaller the data flows, the more stable the connections.
6. Is there a way for you to detect whether there hasn’t been any change from last time to not go through the updating process?
7. Would there be a way to have different schedules at night and during the day? Etc.
This is a great idea and already on the roadmap! We are planning on releasing this feature in Q3 this year.
I hope this small compilation helps!
Any questions? Reply below, thanks!