Monday, August 10, 2009

#0022 Workflows vs Pipelines

040130-N-9693M-020 U.S. Naval Academy, Annapol...

From One to Many  
In this, part 4 of our series on pipelines we will look at how a pipeline and a workflow are connected and how they differ from one another,  an understanding key to talking about and discussing pipelines.
In our previous posts, we explored the basic ideas of pipelines.  In post #0018 Understanding Pipelines, we de-constructed the three classes of CG pipelines: production, material and approval, or task, data and meta-data.  We also looked at how the word "pipeline" may have entered CG production management jargon from its roots in CG software and hardware.  In post #0019 The Pipeline -Form Follows Function we examine these three classes in greater detail and discuss how the form of a pipeline is dependent on the mission and personnel available. 

In part three, #0020 The Technology is Not the Pipeline, we differentiate technology pipelines as found within software and hardware tools from production pipelines.  We also introduced the three dimensions of pipelines: personnel, tools and procedure.    We continue now to define the scope and nature of CG pipelines by dealing with a common area of confusion, the difference between a workflow and a pipeline.  While this was dealt with to some degree in bog #0020 using the analogy of a fry cook and a baker,  I wish to refine the concept for us.

As you may recall, in the analogy both the fry cook and the baker operate from a recipe and they both follow a more or less linear progression of steps.  To some this may seem that their work is structured more or less the same.  But there are two important distinctions.  First, the baker's work can be divided into meaningful separate tasks.  Second, the baker's work can be divided between two or more people. (Now, one can argue that a fry cook's work can be divided between prep and action, and may be in some restaurant, but go with me here.) 

A workflow is like a recipe.  It describes the sequence of steps within a task or job required to get the work done.  The complexity of the workflow can be small or high.  It can be divided between people or not.  It can be divided into meaningful and separable units.  So when should we call a workflow a pipeline?   Are all pipelines workflows?  Certainly.  Are all workflows pipelines?  No.

There is little communication advantage to have two words that mean exactly the same thing.  Usually synonyms differ in shades of meaning.  In this case I believe we can define a computer graphics workflow as the steps or recipe employed by an artist using one or more computers and related technology to produce a CG work product.  

This would mean that when the work is done by a single artist, we call it a workflow, and when the work is divided among a team of artists working on successive or parallel portions of a workflow, it becomes a pipeline.  Each artist's portion is that artist workflow.  The whole is the pipeline.

There is a difference between a project done solo and a project done with a team.

As a CG artist, I am a solid generalist, which is not to say I have no expertise, but rather my expertise is general.  Further, I do light editing work and can do full-blown compositing.  A situation common to many artists.  When I do a job comprised of design, modeling, animating, etc and ending with compositing and integration into a edit, I could call this a pipeline, but in reality there has been no transfer of data, no division of tasking, from one worker to another.  SO the aggregate of what I have done is a workflow,  albeit a complex one comprised of a succession of steps.

Likewise, I have done work in team situations.  The smallest team I've been a part of is two, and by dividing the workflow we establish a de-facto pipeline.  For example, one person doing the 3d work and another doing the compositing.  As teams grow, the pipeline gets divided, in 3d the first division is usually modeling and everything else, with skinning and rigging lumped into modeling if the artist has the skills.  Then animation separates off from the modeling of the shaders, textures and lighting.  Animation then breaks into sub disciplines, as does shading, texturing and lighting.   Additional specialists are soon doing rigging, particle dynamics, water dynamics, cloth dynamics and so forth.  Likewise, compositing can differentiate into disciplines such as key extraction, rotoscope, cleanup, wire removal, rough or layouts, final effects and so forth.

 Image by wabberjocky via Flickr

Mist the Oak Tree

   The Tree of Specialization
The entire workflow can be divided using a hierarchy or tree of specialization, a taxonomy of CG production jobs, with each branch point in the tree a potential decision point for either separating or combining specialties to construct the specialization tree and define the pipeline.

Further, it is important to note that this specialization, while usually thought of as specialization of labor, occurs in all dimensions of the pipeline. Yes, personnel are hired for their specialized skills, knowledge and talent.  In addition, they often may use special tools or use common tools in a special way.  Finally, within their discipline, they are likely to have their own special procedures and policies.

Take the 3d Modeler as a case.  This worker, often thought to be an entry level person, may in fact be a senior artist with years of experience.  He may bring to the craft skills as a traditional sculptor or illustrator or industrial designer.  Within the CG craft, he is likely to use a special tool, such as Z-Brush or Mudbox; or be an expert in the modeling functions and workflows in a program like Maya or 3D Max.   Further, within a modeling department, the modeler often will have specific naming conventions for files and data structures (objects) within the files, and other procedures governing how models are checked-out and checked-in for work and revision control.
    A team project is not necessarily a pipeline
 In case there is any confusion, the fact that a team of artists is working on a project does not make it a pipeline.  What makes it a pipeline is how the workflow is divided between the artists.
Two examples of teams that do not use a pipeline:
First, a basketball team.  Whether working zone or man-on-man, each player has a role and assigned task on the team.    And while one player may have as a primary task to get control of the ball from the opposing team, pass it to another player who moves it down court, and hence to the star who makes the shot while others are guarding and defending, the division of game play among the team players can hardly be called a pipeline.  The players are working in concert, but no player produces anything that is then used by another player as a source.
Image via Wikipedia
Oil drilling rig in the North Sea.
Likewise, a team of roughnecks drilling for oil in the North Sea does not divide their workflow into a pipeline.  Each member of the team may have an assigned task in support of an operation like extending the drilling pipe.  One person may stabilize the pipe, another running the gear that hoists it and another will disengage the pipe from the drill and reconnect the longer pipe.  Again, the procedures do not constitute a pipeline, but rather constitute a team workflow.

The assembly line of Henry Ford is a kind of production pipeline.  What differentiates it from other team workflows is not that each person has an assigned task, but that each person's assigned task produces a component or refinement of the final product.  Further, that person's work is vital to the final product and cannot be omitted.  If the person is absent, another worker must step in or the assembly line will come to a halt.  The difference between Ford's assembly line and a CG pipeline is really rather small.

The defining characteristic hence requires that the workflow be divided into meaningful stages assigned to specialists each producing a component or phase of the final product.
To summarize, the CG pipeline differs from a workflow in that workflows are done by a single person or a team of persons producing a shared result.  A pipeline is a series of workflows, each done by a single specialized person (or a team!) using specialized tools and procedures and each producing an intermediate product of  component part of the whole, until the last worker who produces the final result.
Remember to carefully dry your renderings before stacking!


Anonymous said...

Great article, thanks