#0018 Understanding Pipelines

Deconstructing the "pipeline"

Among your key responsibilities as a Computer Graphics Supervisor may be the design or implementation of a CG pipeline.  Even if you do not design one, you will manage one, and as the pipeline manager your concern will be to keep it running efficiently.  Further, even if you do not design the pipeline you manage, you will be accountable to management to insure the pipeline design and staff capabilities are most appropriate for current needs.

Your task will be compounded by a very simple problem: jargon.  The word "pipeline" is a rather meaningless term.  Like so many terms in computer graphics, it has been adopted from some other discipline and the word crafted to mean something for a specific situation.  You will soon find that there is little literature on the subject and that the word "pipeline" means different things within computer graphics to different people.

Case in point:  I'm sitting across the table discussing an employee's recommendation that we need a "pipeline", a recommendation he's made to myself and my boss, who wants to know why we don't have one.  As I sit there, I'm thinking about how anyone could think our crew of 10-20 artists could have spent the last 12 months making around 1800 vfx shots without a very functional pipeline.  Perhaps, I think, he wants a change in the pipeline he's been part of for 12 months?  So I ask him what he means, and he begins describing an asset management system....

The term "pipeline" may seem to be a term grounded in the antiquity of computer graphics if you've only been around a few years, but it is a rather recent bit of jargon, introduced 10-12 years ago.  The word "pipeline" replaced "workflows", which in turn replaced the terms "cg process" and "cg production phases" around 20 years ago.  The change in terminology has both opened and constricted our thinking about visual effects production processes and the people who make them happen.

The term "pipe" was introduced in computer jargon with the development of UNIX.  A UNIX pipe is an input-output data structure that eliminates the need for program A to write a stream of data to a file that will be read and processed by program B.  Instead, the operating system command line language allowed the insertion of a pipe symbol, which meant, "connect the output of program A to program B", skipping writing the file to save time and disk space.  

A few years later, a young company making a computer optimized for graphics, Silicon Graphics, designed and sold graphics processors using a "pipeline" concept.  Essentially, the graphics engine developed by SGI encapsulated in hardware, at a great speed advantage, the UNIX concept of piping the data through a series of simply written modules making a series of small operations into a significant result.

Over time the concept of moving computer data through a series of software modules would get translated into a work process involving progressing the work product through a series of specialized workers (who happen to use node-based software and other technologies all based on the software pipe concept).  And so "CG process" became "CG workflows" and these became "CG pipelines"....

Back to my employee meeting.  His answer got me to think about pipelines differently.  For some time I had been thinking of the pipeline in a very narrow sense, as a type of CG  work-flow or description of the CG process.  Suddenly I saw that a pipeline could be the flow of any kind of data, not just the flow of our work product, and that the CG pipeline we used consisted of a multitude of different classes of pipelines that touched one another in places.  In an instant, concepts I was thoroughly familiar with became redefined and reorganized.  Like parallel universes, pipelines coexisted within my production environment in different dimensions of perspective.

  

    The Three Pipelines

The Production Pipeline

The traditional way of thinking about a pipeline is the production pipeline.  This is essentially modeled after post-industrial assembly systems where workers specialize in a given task.  The assembly line is a such a system where the work product moves past a series of work stations at a fixed speed, forcing each specialty to be constrained to a fixed duration and narrow set of skills.   Computer graphics is unlikely to achieve the efficiency of Henry Ford's automated line, but the concept of specialization has been embraced. We can take advantage of the phased nature of CG production --model construction, motion construction, lighting, shading, rendering and compositing to break up the work.  The degree of specialization of a pipeline is a subject for some study: not all work missions will support the same production pipeline profile.

The Materials (Or Data) Pipeline

Every CG environment involves the delivery of a product, a shot, a sequence or a complete show.  Along the way assets are gathered and intermediate assets or work products are constructed.  Assets used as inputs may include footage, stills, and reference materials.  One person's output becomes the next person's input.  Hence the pipe-line.  Managing the flow of materials through a shop is an important sub-pipeline, in fact it's really a corollary pipeline.  

The Approval (Or Metadata) Pipeline

A key material in any pipeline comprises the aggregate of direction and instructions about a particular work product (again, a shot or element of a shot at the finest level).  Managing this data involves collecting, organizing and distributing the information to the right persons.  Because it involves a decision loop, the approval pipeline is itself a looped pipeline with an iterative nature.  

Other Pipelines

Adjacent to these may be other pipelines, but these are the essential three.  The first is labor; the second is the data used  or made by labor; and the third is the metadata, the data about the data.  Any other pipelines are special cases of one of these three.  For example, a film-out pipeline is a special case of a production pipeline.  A rendering pipeline is a part of the production pipeline. A material check-in procedure is a sub-set of a materials pipeline.  A sign-off procedure is a part of an approval pipeline. A payroll system is a meta-data pipeline that collects and processes information approving compensation for the work done.

The Integrated Pipeline

The degree that these three pipelines are brought together can determine how synergistic the entire operation is.  In many operations, these three pipelines exist more or less in isolation.  In some operations, the piepline is highly computerized.  In others, the data may be managed through slips of paper, whiteboards or your gray matter as supervisor.  While products exist to help manage the data and meta-data pipeline, these depend on a solid foundation of procedure and conventions.

The Best Pipeline

The question is, what is the best pipeline?  The answer is simple: the most appropriate one.  Some people hear a bit of buzz about the pipeline at shop ABC, and think that everyone should organize the same way.  Or they hear some marketing hype about a wonderful asset management system or project tracking application, and they are all hot for it.  Your job as a CG Supervisor is to help management sort through the noise and recommend the most appropriate pipeline to satisfy the short-term job at hand and the long-term realities of your company.  A pipeline that may be perfectly beautiful and elegant is worthless if it does not fit your company's needs.  Analyzing those needs and weighing your options is another story.

 

Thanks for joining us today.
Now go make some pretty pictures!