Custom data backup in Bitrix24 is a pain reliever for businesses with many years of history and large data sets

The most valuable thing that Bitrix24 has is information: about your customers, deals with them, and completed profitable orders. It is for this reason that they once invented TsRM, which should help businesses develop. However, if you use Bitrix24 for many years, your system will definitely lose performance and the main reason for this is the increase of the database. Next, we will talk about a real successful case of data backup in Bitrix24, which saved all the client's data and overcame the problem of system load. This will be relevant to everyone who works with any edition of Bitrix24 on-premise, because they are all based on a common logic, which means they have the same problems. Read to the end and you will definitely learn about all aspects of the solution you need.

Disclaimer

Before you start reading this article, we have to say: the described situation with the work of Bitrix24 on-premise is general, but our solution is not universal for each individual case. The project would not be successful if:       

• Avivi's experts did not thoroughly study the client's business processes;

• The work was not practiced as well as possible and changed radically;

• We wouldn't be Avivi , a professional solution development company for Bitrix24 that knows everything about the product, its features and capabilities.

So if the speed of your Bitrix24 decreases over the years, providing a comprehensive and effective solution will be possible only if you analyze your business processes in detail. But you can fix the situation in 99.9% of cases.

Briefly about the prerequisites of the decision

Our client has been using Bitrix24 for about 10 years, 6 of which at the time of writing, Avivi has been providing technical support for the system. During this time, CRM has accumulated more than 7,000,000 records only for the Entity "Lead", which is quite a large number by the standards of Bitrix24. The on-premise edition underwent many customizations specifically for the customer's business processes and was updated to the latest version, but over time the problem of speed came to the fore: every request from employees to CRM caused an increasing load on the system. For users who had more than 100 in the company, it looked like a long delay between action and response — up to 7 minutes of delay after any operation. We will remind you that usually more than 30 seconds of delay results in disconnection with the server with error code 504. However, it happened more tragically at the software level, when the entire RAM of the server part experienced constant and long-term overloads.

Next, you will learn why neither updating the on-premise edition of Bitrix24 nor increasing server capacity helps in this situation. However, our client made a difficult decision to delete a large part of relatively old contacts in order to reduce the size of the database. Fortunately, after studying the situation, Avivi experts managed to find an alternative solution. After all, even "old" leads that applied to the company at the beginning of the implementation of Bitrix24, sometimes returned and brought income. So, thanks to the deep understanding of business processes and principles of work of Bitrix24, our developers offered customization that could solve all urgent problems. And it worked.

Reasons for Bitrix24 speed reduction due to database growth

For the best understanding of the essence, we offer a simple explanation of the operation of the Bitrix24 database. Other CRMs work in almost the same way, because in a narrow sense, all these systems are a set of various data related to each other. The Bitrix24 database consists of many tables that are filled unevenly and depending on the direction of the company's work. However, if you are actively working with a huge customer base, it is natural that the most data will go to those tables that contain information about leads , contacts, companies, subsequent deals with them, etc.

However, some tables grow unevenly over time, but the number and relationship between different table fields is important. So when an average user of Bitrix24, for example, a company manager, opens the Leads section of CRM, the system generates a series of queries to the database that should find, read and display the information the person needs. Requests are processed by the server's random access memory (RAM) and processor (CP), the resource of which is always limited. The system reads all data fields in the database tables and includes unevenly extended tables. In our case, some tables with millions of records accumulated by the client amounted to more than 60 GB of permanent memory. So now you can imagine the computational effort the system must allocate to process just 1 request from 1 user for just 1 entity.

Why powerless updates?

Some believe that the upgrade version of the Bitrix24 edition can affect the performance of the system. In general, yes, because each new version is the correction of previous errors and the introduction of new technologies that improve certain aspects. However, the principle of working with the database remains unchanged, so in our case nothing has fundamentally changed. The logic of working with the Bitrix24 database is practically no different from other products and is already optimal for the world of digital technologies.

Why powerless server upgrade?

Any server has its own maximum capacity, which even this complex equipment cannot exceed. The issue of memory overload has always faced IT specialists, but today its solution has reached its maximum potential. So a banal increase in RAM, let's say double (from 32 to 64) cannot significantly affect the situation. Even a larger amount of memory will be instantly filled with a request to a huge database and the system will "get down" as before. Our specialists calculated that in the client's case, increasing the RAM would speed up the response by a maximum of 1 second, which is extremely high by technical standards, but practically makes no sense to the user. In addition, the increase in capacity significantly affects the cost of the server. So the illusory success in shortened long-term responses is not comparable to economic indicators.

Solution from Avivi

There is a long-known rule: "If you are not able to change the situation, change your approach to it." After a detailed study of the client's business processes, our team came to the conclusion that the old adage really makes sense. So now the developers faced an important and responsible task — how to customize the system in such a way as not to harm the general logic of Bitrix24. In general, this is a challenge facing all developers of ready-made products, which Bitrix24 is. After all, creating any new separate solution is not as difficult as trying to implement in the code the expansion of opportunities for someone else's development. Thanks to expert knowledge and many years of experience, our team succeeded in this.

First of all, new tables were created for recording lead data with the working name "Reserve", duplicating the structure of the original tables (in fact, these are several tables, but these are already technical details that are not of particular importance). From the old large database, data about irrelevant leads with which there was no activity in the last few years moved here.

The next step was to intercept requests in the automation processes of the company's existing work flow to the live table and distribute the load on the data. In this way, the work of the entire Bitrix24 ecosystem is reproduced in the usual mode, but without recourse to a large volume of backup data. It is worth noting that the new approach does not contradict the general logic of the system's operation and takes place in the usual cycle.

Using a live table reduced query response time from 7 minutes to a few milliseconds, because now you don't need to iterate through millions of related records to process data. This data is quite sufficient for the active work of the company, and in addition, there remains a large amount of resources for expanding the table in the future. This is due to the peculiarity of the business, since the client works with "hot" leads, the "usefulness" of which is approximately one year.

Again, we have to return to the statement that the key to the implementation of the project was a detailed study of the client's business processes. The "Reserve" stores data on leads with which no action is currently planned. These people may re-apply to the company, mainly with payment for services. In this case, custom APIs intercept payment notifications and check with the "Reserve" table. If the ice is there, it is transferred to the live table.

Results

As you can see, the solution from Avivi really achieved two goals at once:  

• Ensures an optimized process of processing requests without excessive load on the server;

• Stores the most valuable thing for CRM and business in general — customer information.

The massive database that has been formed since the first days of using Bitrix24 "rests" in storage and only occasionally responds to requests through the API or targeted requests from employees when they really need to find information about leads from many years ago.

It would be fair to say about the future additions to the task, namely the reverse transfer of data from the live table to the "Reserves" in case of a change in the status of the lead (for example, "deleted") or the expiration of the "expiry" period. However, for now, all entries in the table remain relevant and there is no urgent need for this.

So if your Bitrix24 does not please you with speed due to the accumulation of information in the database or for other reasons, do not rush to delete something. Let Avivi experts study your business processes and, we are sure, a custom technical solution will definitely be found!