So a prep into the database followed by running on queue restart has been more reliable.
I ran into problems on some runs after 6-8 hours of run time. Pantheon does have issues with systems errors if left to run a batch job for extreme runs.
I’ve run this process for several hours at a stretch. the Queue UI module provides a simple interface to run those on a batch job.
#Pantheon drupal code
The example code includes comments about generating a queue worker to run on cron or as another batch job. It can also be used as the base to create more complex objects. If you need to add a large data set to an existing site that’s used for reference data or something similar. The example code just dumps this all into a database table. $queue = $this->queueFactory->get(‘example_pantheon_loader_remap’) If you're setting up for a queue you include something like this. $row_id = $this->database->insert('example_pantheon_loader_tracker') Each pass we process 100 lines, if you have to do something complex To get us started the form includes this managed file: If you need to do more than load the data into the database directly (say create complex entities or other tasks) you can set up a second phase to run through the values to do that heavier lifting. From there the batch service uses a bit of basic PHP file handling to copy data into the database. The first is a form that provides a managed file field which create a file entity that can be reliably passed to the batch processor. My code samples below provide two classes to achieve this. The general concept here is that we upload the file and then progressively process it from within a batch job. It could be adapted to a traditional batch job, but I like the clarity the wrapper provides for this discussion. One quick note about the code samples, I wrote these based on the service-based approach outlined in my post about batch services and the batch service module I discussed there. While you can actually load the data raw, you can also load each record into a table or a queue to process later. It does not have to represent the final state of your data. For this solution you need a file you can load and parse in segments, like a CSV file, which you can read one line at a time. Since Drupal’s batch system was created to solve this exact problem it seemed like a good place to start. When I first estimated the process the running time was over a week.
We were looking at dong about 50 million data writes for the project. The minimize user impact I started to play around with solutions that would allow me to ignore those time limits. I recently needed to do a very large import of records into Drupal on a Pantheon hosted site.
Or the data load takes too long, or happens too often, for that to be practical on a regular basis. But sometimes those things are not an option. And can afford some additional project overhead. And have the time to setup a process to mirror changes into your temporary copy. That’s fine if you can afford to freeze updates to your production site. While they are large enough for the majority of projects, large projects can have trouble.įor data loading their official answer is typically to copy the database to another server, run your job there, and copy the database back onto their server. These include process time limits and memory limits. But to make their platform work and scale well they have a number of limits built into the platform. Pantheon is an excellent hosting service for both Drupal and WordPress sites.
0 kommentar(er)
