What kind of flowchart provides an overall view of the inputs processes and outputs of a system?

V.F. Higher Level DFD

The level 2 DFD expands the processes in the level 1 diagram. The level 3 DFD is an expansion of the process in the level 2 DFDs. The level 4 DFD is an expansion of the process in the level 3 DFDs. And the level 5 DFD is an expansion of the process in the level 4 DFD etc. Levels stop when the analyst feels enough detail has been reached. All processes may require different levels of abstraction. In other words, eventually we will decompose the DFD diagram to a point where the DFD cannot be decomposed any further. This point is usually reached when the processes themselves only describe simple tasks. When we reach this point, the contents of the process “bubble” need to be described. In Fig. 10, such a point has been reached.

IV.A.3. Entity–Relationships Diagrams

DFD, despite its name, is a logical modeling tool focusing mainly on processes without emphasizing data relationships and flows. Entity–relationship diagrams (ERD), proposed by P. Chen in 1976, are a conceptual modeling technique emphasizing data entities of an organization and their relationships, complementing, thus, DFD. In an ERD, an entity represented by a rectangle is an external or internal element or a transaction of an organization, for example, a customer or a sale. A relationship, illustrated with a diamond, refers to the connection of two entities, such as a sale to a customer generating a sales invoice. Thus, the identification of entities and relationships offers a comprehensive view of the structure of the database of the current or proposed system and assists in both system analysis and design phases of the SDLC.

Data Flow Diagram

A Data Flow Diagram (DFD) is a graphical representation of the “flow” of data through an information system (as shown on the DFD flow chart Figure 5), modeling its process aspects. Often it is a preliminary step used to create an overview of the system that can later be elaborated. DFDs can also be used for the visualization of data processing (structured design) and show what kind of information will be input to and output from the system, where the data will come from and go to, and where the data will be stored. It does not show information about the timing of processes or information about whether processes will operate in sequence or in parallel.

It is common practice to draw the context-level data flow diagram first, which shows the interaction between the system and external agents that act as data sources and data sinks. This helps to create an accurate drawing in the context diagram. The system’s interactions with the outside world are modelled purely in terms of data flows across the system boundary. The context diagram shows the entire system as a single process and gives no clues as to its internal organization.

This context-level DFD is next “exploded” to produce a Level 1 DFD that shows some of the detail of the system being modeled. The Level 1 DFD shows how the system is divided into subsystems (processes), each of which deals with one or more of the data flows to or from an external agent, and which together provide all of the functionality of the system as a whole. It also identifies internal data stores that must be present in order for the system to do its job and shows the flow of data between the various parts of the system.

Software Design Diagrams

Software analysts and designers use a variety of diagrams, not for variety's sake but because different types of diagrams highlight different aspects of a system. For example, data-flow diagrams show the functions of the system, state-transition diagrams show the timings in the system, and entity-relationship diagrams show the data relationships.

The diagrams software developers use will also depend on their development culture—in other words, what most people are familiar with, what your spec-writing guidelines say, whether the software company does mostly transactional systems or object-oriented systems, and so on.

The three most common software diagrams and a few of their variations are listed in this section. For information on other types of software diagrams, see Resources.

Data-Flow Diagram

The data-flow diagram (also known as the DFD, bubble chart, process model, business-flow model, workflow diagram, or function model) is good for displaying the functions of a system but not good for modeling databases or time-dependent behavior (Figure 13-30).

FIGURE 13-30. Data-flow diaqrarn.20

The data-flow diagram (DFD) pictures a system as a network of functional processes connected with flows, plus occasional collections (called stores) of data (Yourdon 2001, pp. 1–3).

Primary Symbols

Processes or functions are shown as circles or sometimes as rounded rectangles. A process shows a part of the system that transforms inputs into outputs. The label should be a word, phrase, or short sentence that says what the process does—for example, “Find information on DFDs.”

Flows are shown as arrowed lines (either straight or curved). The label should say what kind of information or item moves along the flow—for example, “Web link.”

Stores (places where data are stored) are shown as two parallel lines or an open-ended rectangle. Not all systems have stores.

The label is generally the plural of the name of the items carried by the flow into and out of the store—for example, “Web links.” (This implies that the same items go in and out, so the flows into and out of the store will have the same labels.)

Terminators, external entities or people with which the system communicates, are shown as rectangles. The label is the name of the terminating entity (for example, “Web application book”), person, or group of people.

Manage Complexity by Breaking the Diagrams into Levels

Because the diagrams can become as complex as the systems they describe, analysts have a method for breaking up a diagram into manageable pieces: They create levels. They start with one “context” diagram that shows the entire system at a glance, numbering the major functions—see, for instance, the numbered bubbles in Figure 13-30. Then, subsequent diagrams use subnumbers that refer back to the context diagram—“1.1 Receive orders via web site”; “1.2 Receive orders via phone”; and so on (Yourdon 2001, pp. 15, 19–20).

Entity-Relationship Diagram

The entity-relationship diagram (also known as the ERD or E-R diagram), is good for describing the layout of a stored-data system. It is not good for modeling functions or time-dependent behavior (Figure 13-31).

FIGURE 13-31. Entity-relationship diaqrarn.21

The entity-relationship diagram (ERD), because it is relatively simple and familiar, is a good communication tool. It can be shown to:

•

Executives who ask about the data used to run the business.

•

Systems analysts who need to see the relationships between data storage systems.

•

The data-administration group that maintains the global, corporate-wide information model.

•

The database administration group that manages the corporate databases and implements changes.

Primary Symbols

ERDs have two main components: object types and relationships. Object types are shown as rectangles. An object type represents a collection or set of objects in the real world. The label is a noun or name, usually singular.

Note that objects in an ERD can correspond to stores in a related DFD. For example, if there is a CUSTOMER object in the ERD, there should be a CUSTOMERS store on the DFD.

Relationships are indicated with lines (or diamond shapes). One-to-one and one- to- many relationships can be indicated using single-headed arrows (1 to 1) and double-headed arrows (1 to many). Direction (from Object 1 to Object 2) can be shown with the arrows as well. A required relationship can be shown with a short line; an optional relationship can be shown with an open circle.

Variations

Unified Modeling Language Logical and Physical Data Models

Unified Modeling Language (UML) logical and physical data models are good for modeling object-oriented databases. The diagrams can indicate inheritance as well as a wide variety of relationship types.

The diagrams can be used to show either logical (Figure 13-32) or physical models (Figure 13-33).

FIGURE 13-32. UML logical data model.22

FIGURE 13-33. UML physical data model.23

•

Logical data models (LDMs) show the logical data entities, the attributes describing those entities, and the relationships between entities.

•

Physical data models (PDMs) show the internal schema of a database, including the data tables, their data columns, and the relationships between tables.

The visual components are rectangles, with an area at the top for the object name, and lines that show the relationships between tables and sometimes also inheritances. For more information on notation and analysis, see the books listed in Resources.

State- Transition Diagram

The state-transition diagram (Figure 13-34) is good for showing a system's time-dependent behaviors. Originally designed for real-time systems such as process control, telephone switching systems, high-speed data acquisition systems, and military command and control systems, state-transition diagrams are now used whenever timing might be an issue—for example, if thousands of terminals might hit a database at the same time or when activities occur in flurries, as shown in Figure 13-35.

FIGURE 13-34. State-transition diagrarn.24

FIGURE 13-35. Complex state-transition diagrarm.25

Approach Considerations

The content to be processed is taken directly from the data models, Preliminary Taxonomy, or any other list of content or BIRs.

The actual modeling can be traditional data-flow diagrams, swim lanes, or any other technique that is acceptable. Just agree on one method beforehand. You will need to use SMEs or facilitated sessions. If there are a large number of business process opportunities, you may even want to consider doing a batch of preliminary processes, then selecting the ones deemed more important. Process modeling can be time consuming.

Once the basic versions are completed, they need to be optimized. This means interpreting the Metrics and Requirements data and mapping to the various characteristics. For example, a process to develop a cross-sell strategy for strategic customers may be perceived as an event to take place as soon as the large customer is known. But if the metrics and BIRs to support the processes are identified as being OK for a monthly provisioning, some clarification is in order.

Once the models are optimized, develop and conduct management-level presentations to review them, go over the benefits related to the new processes, and obtain recognition of the opportunity and guidance on how to proceed.

3.1 Compounding component

Structuring the environment as components of a data-flow diagram is well suited for the implementation of modules needing a single access to the information derived from the trace. An example of such a module could be one that computes the CPU usage of nodes—for each received thread state which is not a blocked state, the module adds the duration of the state to an accumulator corresponding to the thread's node; it is never necessary to access the state again. Another example is a passive visualization module that, for each received entity, displays a corresponding visual representation that cannot be interrogated or changed.

However, if the module receiving the data is interactive, it needs to access the data objects several times. This is the case of a visualization module that gives the possibility to inspect the displayed objects or provides a historical vision of a trace (allowing one to display previous portions of the trace). If such a module receives the data objects traveling on the data-flow graph independently, it should either store the objects or have to query them again each time they are needed. The first solution results in added complexity of such modules, to manage a large volume of data, as well as data replication if more than one module of this type is used. The second solution results in added computation costs to read and simulate the trace several times.

To overcome this contradiction, a new type of component was designed and implemented, to produce compound objects. The compounding component encapsulates the elementary objects produced by the simulator into a single object which represents the current observation window of the parallel program execution. This compound object is produced by the compounding component and consumed by the following components in the data-flow graph (see figure 2). Each elementary object input by the compounding component is linked to the compound object. At regular intervals, if the compound object has been changed, it is put in the data-flow graph, so that the following modules of the graph can take the changes into account. All the accesses of the other components to the elementary objects are done through the compound object, according to a well-defined protocol. This protocol defines how to obtain global information relative to the current observation window (number of nodes, maximum number of threads in each node, etc.), get the elementary objects in a temporal sub-window, get more information concerning an elementary object and ask for the inspection of an elementary object. This way, all the complexity of storing and accessing the large quantity of data generated by the trace is isolated in the compound object.

Besides simplifying the construction of data-consuming modules, centralizing access to the data has some other advantages for the construction of filters (see section 3.3) and for the memory management by the controller component.

Decomposition Matrix

The decomposition matrix tool generates either business process or data flow diagrams. It does this using an algebra for design decomposition that Mike Adler published in the 1980s.1

A feature of the decomposition matrix is that it does not look at all like a business process or data flow diagram. Business process diagrams, for example, are a great way to design a workflow. The problem for most of us, however, is that if we are familiar with a given workflow, it is often difficult to see how it could be significantly different. We all tend to repeat or recreate what we know. The decomposition matrix, however, requires us to only think about inputs, outputs, and how they relate to each other. The diagrams are generated for you based on the matrix of inputs, outputs, and relationships.

A significant issue when making any systems change, particularly in large organizations, is getting agreement on what the changed system should do. This compounds the situation where it is often difficult to see how the changed system should be. Not only might individuals have a difficult time thinking of how their workflow could be different, there might be entirely different views of the workflow in different parts of an organization. A tool like the decomposition matrix can be a way to address different views within an organization by getting people to only think about inputs, outputs, and how they relate to each other.

I have the decomposition matrix tool implemented on one of my websites.2 It is free to use. It can be used in a group setting if you have a computer with an Internet connection hooked up to a projector.

Figure 10.1 shows a decomposition matrix of inputs, outputs, and relationships. It allows you to discuss detailed issues one at a time instead of trying to juggle multiple issues all together in a design. You only need to make a series of binary decisions. Such a decision is whether a given input is related to a given output. Sometimes that can generate a great deal of discussion and bring out design issues not previously mentioned. The decomposition matrix assembles these simple decisions and generates a decomposition that might help you with your design process.

The tool on my website generates either business process or data flow diagrams. Most people are familiar with business process diagrams. The data flow diagrams are a way to get at the decomposition of services in an SOA. The decomposition matrix has a specific definition of atomicity. Atomicity generally means that a business process cannot be decomposed further (see page 17 for a general definition of atomic services). The specific definition of atomicity used by the decomposition matrix is that a business process task or a data flow process is atomic if every input relates to every output in the decomposition matrix. In other words, there are check marks in every box of the matrix. Atomic tasks and processes are an important aspect of the incremental SOA analysis.

It is possible that the decomposition matrix might give you some new ideas or help you get past a sticking point in your design process. In that way, it acts much like having another designer in the room. The decomposition matrix is not a design methodology. It is meant to be a design aid. You can use it with whatever methodology you prefer since it is just another “designer” in the room.

The next section provides an example of how this tool works.

Business Process Diagram

To illustrate how the decomposition matrix works, I will use an example from a series of blog posts that start at http://www.designdecomposition.com/blog/?p=6. This example uses a set of inputs and outputs for a travel coordinator. Using those inputs and outputs, the decomposition matrix tool will generate a business process diagram.

This example is from the first edition of this book. The idea that a VPA—like the one in the story about C. R.’s business trip—could make all travel arrangements was not considered when I wrote the first edition. Nevertheless, making travel arrangements is an almost universally understood process so I decided it is still a useful example for the decomposition matrix.

The inputs and outputs in Figure 10.1 should be familiar to most people who have taken a business trip. They involve finding airline flights, a rental car, and hotel rooms for a set of travel dates along with making the reservations and obtaining driving instructions. Figure 10.1 shows this decomposition matrix.

You need to consider the relationship between only one row and one column at a time when using the decomposition matrix. These are the binary decisions mentioned earlier. For example, you could describe the relationship of the first row and first column as “the input of travel dates and locations that occurs before or concurrently with the output for a flight availability request.”

The portion in italics is an example of the type of phrasing you should use. You may read across the row or down a column using the italicized phrasing.

Considering just one row and one column at a time makes it easier to work with larger designs. There is no need to try to keep the entire design in your head. You just need to think about each relationship one at a time.

Arranging flights involves using the travel dates and locations to request a list of available flights. Sometimes you may need to make multiple requests with different flight times or you may make requests to multiple airlines. Figure 10.1 shows this with a check mark in the second row, flight availability response, and first column, flight availability request. The third row, flight reservation response, is not checked in the first column, because you cannot have a response before a request.

The fourth column shows the inputs that occur before or concurrently with the input to a car rental reservation request. Before making a reservation request, you need to know that cars are available for your travel dates and locations. You also need to know if flights and hotel rooms are available. You do not, however, need to reserve a room before a car. On the other hand, car rental agencies often ask for a flight number at the time of rental. So there is a check mark in the third row, flight reservation response, for the fourth column. This occurs before or concurrently with the output for a car rental reservation request.

The generated business process diagram is shown in Figure 10.2. The diagram uses a subset of the business process modeling notation (BPMN).3 The tool does not generate labels for the tasks. I have added task labels to this diagram.

Figure 10.2. Generated business process diagram.

There are a couple of ways the generated diagram can give you hints that there are problems with the check marks in your decomposition matrix:

n If you have had trouble coming up with any of the labels, that could be a hint that the inputs and outputs might not have the correct check marks or perhaps an input or output was overlooked.

n If the diagram is confusing, that is a hint that the check marks might not be correct. An example of something confusing is a request for something coming in after its related response.

You can “play” with the inputs and outputs to see what happens to the generated diagram. This is not a complete design tool. At some point you may want to transcribe a generated diagram into your design tool, much like you would if you used a whiteboard.

Data Flow Diagram

The next example generates a Web services API or services interface layer for legacy systems. Figure 10.3 shows the decomposition matrix. The inputs are from some type of legacy system. Some of the possible outputs are also shown in the decomposition matrix. It is obviously simpler than the real world, but it serves as an illustration of how the tool can be used.

Figure 10.3. Decomposition matrix for services.

You can phrase a relationship in Figure 10.3 as “the input of invoice is used directly or indirectly for the output of payments.” The italicized portion of the phrase is important. Note that this is different from how relationships are described for business process diagrams. In this case, we are dealing with data flow and not the sequencing that business process tasks require.

Figure 10.4 shows the decomposition of services based on the matrix.4 The processes have been labeled. Just like with the business process diagrams, the tool leaves labeling up to the user. Again, if it is difficult to label a process or if the diagram is confusing, that is a hint that the inputs and outputs may not be complete or that some check marks are missing.

Figure 10.4. Decomposition of services.

The top-level processes in Figure 10.4 represent the Web services API or service interface layer. Some of the top-level processes have multiple outputs. This indicates that the input parameters will need to specify the XML tags (in this case) to include in the output. Such input parameters are not shown in data flow diagrams, but they will be needed when you design the services. Any data flow diagram shows only the flow of data and not the control input parameters.

The services below the top level are reusable components that have been factored out. Depending on your implementation, you could implement them as services or as library code components.

Just like with the business process decomposition, this tool allows you to “play” with inputs and outputs to see the effects. At some point, you will want to transcribe the decomposition into your design tool.

Provide Filtering Options

For diagrams with lots of data (for example, a data-flow diagram for all of the company's actuarial tables), users may need to isolate one or two datasets from the dozens or hundreds that could appear on the window.

To filter a diagram, you may be able to adapt whatever filtering interface you already have in your system. Chapter 6, Data Retrieval: Filter and Browsing, covers filtering.

When there are many elements or when the diagram is used to analyze problems, it may make sense to provide a query-on-query option. In other words, rather than asking users to run new queries each time they have new questions, let them refine the search starting from the current set of data.

TYPES OF PROCESS MODELS

One difficulty in trying to describe activity modeling is that there are so many different ways to do it! To be sure, the underlying concepts are pretty universal, but you would never know it to look at the variety of techniques available.

The first type of process model in general use was introduced almost simultaneously by Tom DeMarco [DeMarco 1978] and by Chris Gane and Trish Sarson [Gane and Sarson 1979]. This was the data flow diagram, which provided a means of representing the flows of data among processes, external entities (the ultimate sources and destinations of data), and data stores (where data could reside temporarily between processes). It included the ability to decompose processes into more detail. In addition, the authors made a distinction between the physical data flow diagram and the logical data flow diagram. Mr. DeMarco extended these ideas to describe the process of moving from a current physical data flow diagram to a logical data flow diagram to a new physical data flow diagram.

Mr. DeMarco was a little fuzzy, however, as to how one distinguished a physical data flow diagram from a logical one. It was not until James McMenamin and John Palmer published Essential Systems Analysis [McMenamin and Palmer 1984] that this process was formalized. Messrs. McMenamin and Palmer came up with the idea of the essential data flow diagram. Its characteristic is that it consists of essential processes, which are defined to be the complete responses to an external event. Essential processes, in turn, are subdivided into fundamental processes, which carry out the work of the enterprise, and custodial processes, which maintain the data required by fundamental processes.

In the early 1990s the subject of business process re-engineering became popular and resulted in the rise of something Geary Rummler and Alan Brache called a process map [Rummler and Brache 1995]. These are similar to physical data flow diagrams, but they organize processes according to the departments or people who perform them. They focus less on the data being communicated than on the sequence of tasks expressed in common language, and instead of being concerned with physical and logical data flows they simply reflect the structure of tasks that currently exist (“as is”) and that are planned as the result of the analysis (“to be”).

Shortly after this, use cases became popular in the object-oriented world. This technique was originally described by Ivar Jacobson [Jacobson 1992] and then elaborated on by, among others, Alistair Cockburn [Cockburn 2000]. These are also similar to data flow diagrams, in that they have communications between external entities (here called “actors”) and processes, but the assumption is that the processes involved represent computer systems (typically shown only as a single process representing the entire system). The content of data flows are not documented, and rather than being decomposed into lower-level detail, these details are simply described in text as “steps”. There is no notion of storing data in intermediate data stores.

Meanwhile, back when Ms. Sarson and Messrs. DeMarco and Gane were inventing data flow diagrams, the federal government was embarking on its Integrated Computer Aided Manufacturing (ICAM) project. This was an approach to designing and building aircraft. Out of this work came some modeling techniques that were later codified by ITT as a Structured Analysis and Design Technique (SADT). In 1981, the Air Force recognized the value of SADT and requested a version for the Air Force. This was the source of the IDEF series of modeling techniques.*

“The full IDEF set (IDEF0, IDEF1, and IDEF3—later expanded to include IDEF1X, IDEF4 and IDEF5)** was designed to support systems analysis, where a ‘system’ may be a computer system, a non-computerized process, or an entire enterprise” [Feldmann 1998, p. 7]. IDEF1 is a general approach to modeling data, whereas IDEF1X specifically addresses the design of relational databases. IDEF4 is a way of looking at object-oriented design. IDEF5 provides a method that assists in the creation, modification, and maintenance of ontologies.

IDEF3 is an approach to describing flows in terms of decisions made, whereas IDEF0 is for simply laying out an enterprise's functions. According to the IDEF web site, “IDEF0 is a method designed to model the decisions, actions, and activities of an organization or system” [KBSI 2005]. In this sense, the idea is that it is modeling functions rather than processes. The same site describes IDEF3 as providing “a mechanism for collecting and documenting processes. IDEF3 captures precedence and causality relations between situations and events in a form natural to domain experts by providing a structured method for expressing knowledge about how a system, process, or organization works” [KBSI 2005].

In 1998, Clarence Feldmann published The Practical Guide to Business Process Re-engineering Using IDEF0 [Feldmann 1998], which is a very clear description of IDEF0. It discusses “actions”, however, which leaves the subtler distinctions between function and process unresolved.

IDEF0 is very similar to data flow diagramming in its discipline, although it does not explicitly document the external entities or data stores involved. It models the logical structure of each process, but it includes the ability in each case to add an arrow describing the mechanism used for the process. In addition, unlike any other technique it distinguishes between data used by the process and controls that affect it and trigger it.

One of the model types provided with UML is a process model [Rumbaugh et al. 1999]. This is organized like a BPR process model, in terms of who is doing the processing, and it does have data stores in the form of “objects”. It also has an explicit symbol for representing processes occurring in parallel.

Table 3-2 shows the techniques just described and their primary characteristics. It is the objective of the metamodel in this book to encompass nearly all concepts revealed by all of these techniques. By necessity, each entity class can have only one name (and many of them come from data flow diagramming), but the intention is to provide a structure that can accommodate all approaches.

Table 3-2. Comparison of techniques.

Metadata Management

Metadata management includes planning and control activities that enable access to definitions, models, data flow diagrams, and other information critical to understanding data and the systems through which it is created, maintained, and accessed. Understanding requirements for quality starts with understanding what data represents. Metadata provides this information. Well-defined, well-managed metadata reduces the time it takes to find and use data. It also reduces data-related risks. If data is well-defined, it is easier to classify and protect appropriately. There is less chance that it will be misunderstood or misused. The opposite is also true; poorly managed metadata can result in confusion about the data itself and can lead to mistakes in integrating, accessing, and using data.