Concepts

Data Ingestion Timeline

What does the data ingestion timeline look like?

Data Ingestion Timeline

We are continually gathering, processing, combining and enriching data. We then update the database and make it available all at once on a daily basis. Here is what the timeline looks like:

How long does it take to update to take a change

1

Day 0: Data updated on a property website

A property website is updated with new data, for example a new price for a unit or a new concession

2

Day 0: Change detected and saved

Our system is constantly monitoring property websites for new data. When the new data is detected, it is added to our ingestion queue.

3

Day 1: The data is combined with other data

We combine all changes from the previous day into a single update. This process involves parsing the data and storing it in a structured format.

4

Day 1: The data is enriched

The data is enriched with additional information. For example, we use our models to extract concessions, data from floorplans, detect outliers and issues, etc.

5

Day 2: For each state, one by one, we update the database

During the morning, we update the database for each state. You can expect the data to be fully updated by 12pm. The new price or concession is now available for all our users.