Toggle migration to QuestDB

Description of Toggle use case with QuestDB

Toggle uses AI & Machine Learning to help investors extract insights on their portfolio & investments. The system distills billions of data points into alerts like “Analyst expectations are turning negative for AAPL; historically, this led to outperformance of the stock.” As you can imagine, this sort of system requires a tremendous amount of timeseries data — prices, fundamentals, sentiment, etc. All of this data is stored as a series and needs to be easily accessible for analysis by our models. It is critical that every step in the process is optimized.

Improvements over InfluxDB

Toggle utilized many databases, including Mongo, Cassandra, and TimescaleDB. After much testing, they settled on InfluxDB, as it had the best performance. As the company was growing, performance started to degrade, and it became expensive to run. They had a small cluster of 4 x m4.2xlarge machines, and memory on all 4 was often at least 80%, hitting 100% a few times per week. Modeling out the future infrastructure spend based on this baseline, InfluxDB wasn’t a viable option as the company scaled.

The process to migrate data from InfluxDB to QuestDB

When evaluating a new solution, Toggle knew that they had to answer the following questions:

• Can we move the data seamlessly and promptly?
• Can we query a sample of our data with at least the response times of InfluxDB?
• Can we ingest new data seamlessly?
• Can we create time series on the fly?
• Can we maintain the performance that we have today after we’ve imported all of our data?

Of all the possible solutions evaluated, QuestDB was the only one that met all of our criteria.

A side by side comparison of QuestDB vs. InfluxDB

• InfluxDB on a cluster of 4 machines (m4.2xlarge with 128GiB of RAM) was averaging a response time of over 5 seconds.
• After a few weeks with QuestDB in production (still with a single machine), the performance averaged 15 milliseconds, i.e., more than 300x faster queries.
• The virtual machine’s stats indicated that the servers were never overtaxed (user: 17%, system: 4%).
• Direct cost reduction (¼ of the machines) and performance improvements meant that Toggle could do much more for less.

The actual data migration was easy with a script to read from one side & ingest in the other. Toggle imported over 600 million data points in a few minutes.