The Problem
PHP is not the best choice for long running processes, and never has been. It has two major shortcomings for this kind of application:
- PHP does not support continuous communication with a browser/client on its own.
- PHP is designed to run and then close, and while some variables can carry over between runs, you start with basically a clean slate for each run.
So knowing those shortcomings and knowing that the application is mostly PHP, I had to come up with a way to make a useful and informative user interface for Vogon Media Server to import media. In the prototyping stage I simply let the whole import run as a single long-running process that only output after the import was done. While this was fine for a developer interface, I’m planning on handing this software over to my parents, who should be able to add their own media to it without assistance.
The Method
A solution to this problem has to achieve two major goals:
- The application must be able to split a task into small repeatable chunks
- Those chunks need to be designed to run as completely independent processes.
Add to this that there are a number of features that might rely on similarly large tasks in the future, and it was obvious that rather than writing a single purpose solution, I needed to craft a method that was modular enough to be applied wherever it was needed.
So the first task was to break down what the process was really doing and then using those overarching prototypes to shape whatever process we come up with.
The import was divided pretty easily into two major sections.
The first section was the initialization. This is where we scanned the filesystem to see what files existed in the requested directory, and set up the classes and includes before we went into a foreach loop through the array our filesystem scan created.
Next inside the loop, we test the file to ensure it’s a file we can import, that it hasn’t already been imported, and finally do things like reading the meta data before creating the needed database entries.
So to restructure this, we’ll keep the two basic phases, but we’ll restructure them so that the section inside the loop is designed to be run as an independent process, doing it’s own initialization as necessary. This is slower, but it gives us more flexibility. Then if we utilize AJAX requests to actually run the loop we can get incremental data back to the user so that it’s less likely that a long-running process will be killed by system resource monitoring software, or that the user will accidentally assume a process is finished when it’s still running.
The Solution
The majority of Vogon projects are written entirely in functional programming. The CMV structure I use means that those functional chunks can be modularly re-used, getting around one of the main shortcomings of that approach. But, in certain situations I favor developer usability over code readability. I don’t like to do it most of the time because classes can end up being the equivalent of code black-boxes: you provide input and get output, but who knows what’s happening inside.
In this case, what we want to do is complex but we want to do it easily and quickly so that developers will be incentivized to use it rather than re-inventing the wheel. For this reason I wrote the ajax_loop_interface as a class file. If you’re looking at the Vogon Media Server repository, it can be found at /main/class/class.ajax_loop_interface.php. The class has one public function, the __construct() magic function, it manages everything else for you.
The idea is this, when the class is instantiated the developer provides some initial options:
- The mode we’re running in (currently it supports two modes, ‘session_array’ and ‘db’)
- Our initialization method. If we’re in db mode this takes the form of an SQL statement whose results we can loop through, in session_array mode this should be a model that returns an ordered numeric key array we can save to the $_SESSION super global and loop through that way.
- The model that does the work inside our loop
- The extension these models can be found in, if any
- The variable name for data passed into our loop model (this defaults to ‘$row’ if you don’t set this value)
- A model that does the after-loop cleanup, if there is any to be done
- And a number of optional options that differ based on the mode (parameters for a prepared sql query for example, or a boolean switch for the offset behavior if you’re doing something that will modify the results of the SQL query, such as deleting values.)
The class then takes those and determines the current state of the loop based on the existence or absence of $_GET variables. It uses the variable “offset” to determine where it is in the process. If offset doesn’t exist we need to initialize our process, otherwise it functions as a marker for where we are in our loop controlling what variables we pass to the provided model. Then, based on what is run the class returns state information and model provided updates as JSON data so that it can be easily processed by JavaScript.
Rather than relying on custom JavaScript implementations, I also wrote a view (/main/view/view.ajax_loop_interface.php) that is designed to be included in other views and contains the required DOM structure, CSS, and JavaScript to create a loading bar and an auto-scrolling message box to keep the user informed.
The intended application is that the developer creates an endpoint that initializes the class with whatever your needed options are, and then creates a view that includes the ajax_loop_interface view and with a ‘route’ option that points at the created endpoint. The two files then abstract the rest of the process so the only thing the developer needs to spend time on is writing the models the class runs.
The Result
This approach ended up having pros and cons, but in my opinion the pros far outweigh the cons, but let’s start with the negative points first.
Doing anything this way raises the complexity, and by running scripts via AJAX you’re pushing yourself further from the immediate output which can make troubleshooting code problems a little harder. Not much harder if you know where to find the XHR requests and responses in your developer tools, but harder.
The options are highly variable based on the situation and the selected mode, so a developer will likely have to have the class file open and be somewhat familiar with it in order to utilize the best options available. This is true of all class files, you have to know what input to use to get good output.
But the positives are pretty strong.
The design of the class means that any long-running loop based process can be adapted with just a little restructuring. This is made doubly true if you utilize the ‘var_name’ option of the class to rename the variable to whatever you were using inside the original loop. Then the majority of the adaptation can be as easy as just copy-pasting and removing excess whitespace characters.
Communicating the current state of a long running process is a big boon both in UX and in getting real-time information back to developers who are still debugging their process.
Structuring loops this way automatically creates some error resilience. Even if an unhandled error or exception occurs during one of the runs, the process as a whole will simply report that error to the user and then run the next item in the loop.
Having modular UI element generators makes the process of enforcing UI consistency much easier than situations where developers are expected to read CSS sheets and other implementations in order to structure their work. There’s a reason OS GUIs usually include an API for common elements that OS uses elsewhere, and by making sure we’re utilizing the visual language of the application we can ensure that a user has a reasonable chance at being able to understand what they’re seeing, even if they’ve never seen this particular interface before.
While the class that runs the loop functions as a black box, the actual work being done are just model files. This means that any developer familiar with how Vogon models are written can easily utilize this class by simply looking at an example and poking around. Since Vogon started out as a prototyping framework I think it’s important to enforce readable structures as much as possible, especially since I have a bad habit of inconsistently commenting my files.
