Verifying claims of Open Systems conformance

Derek Jones
derek@knosof.co.uk
Knowledge Software Ltd
Farnborough, Hants GU14 9RZ
UK

ABSTRACT

Open Systems is the latest wonder attribute for computing hardware and software to have. The desire for vendors to be associated with this term has led to many spurious claims of conformance. Here we look at how users might find out exactly how well products do conform to standards. A checklist of activities is provided that should enable those interested in obtaining the facts to see through the gloss that covers them.

Introduction

Open Systems is the latest buzz word in computing. Both hardware and software vendors claim that their products have this wonderful attribute. Requests, by users, for more information is rarely met with details. In this paper I do not intend to give yet another definition of what Open Systems means or even discuss the contents of the standards documents themselves. Rather I plan to suggest possible questions that users might like to raise with their suppliers, next time they are buying products that associate themselves with the term Open Systems.

The intent of these questions is to enable potential purchasers of products to obtain the facts concerning conformance to Open Systems standards. Conformance is only one of the issues that go into making the final decision in the selection of products. But if it is considered to be anything more than superficially important, then procurers need to obtain accurate information.

The sales claims

Before getting down to the nitty gritty, lets look at some of the typical justifications given for claims of Open Systems conformance.

Perhaps the baldest claim is based on the justification that users want it; "Users wanted to see the word Open Systems on the box so we printed it in the brochures". If you cannot shoot down such claims to Open Systems conformance do not read on, perhaps you should even changes jobs.

The other two common claims do have some, but not much, justification. The first is that the application runs on a conforming platform. The assumption being that to run on such a platform the application must itself be conforming. This is an incorrect assumption. The development compiler, used to build the application, is only likely to check for constraint and syntax errors (one of the many classes on non-conforming constructs defined by standards), since it is these constructs that a conforming implementation itself is required to detect (and must detect in order for a compiler to validate). The fact that an application can be built and executed on one platform does not prove that the it can be built on other platforms. Experience over 40 years of software development has shown that it is impossible to produce any significant applications that do not contain bugs. The same principle holds true for writing standards conforming applications. Mistakes will be made. Standards requirements are not always rigid, they contain many constructs whose behaviour is left up to the implementation. Thus an application that runs on one conforming platform may be making use of behaviour specific to that platform.

The second is a follow on from the above claim. It is that an application which has been ported to many platforms must conform to standards. An application that has been ported to many platforms, is, by definition, portable. Portability and standards conformance are often intermixed. Standards conformance certainly improves portability. The reverse is not true. A portable application need not be standards conforming. The theory behind the idea that portability means conformance works along the lines that because each platform could potentially behave differently, for the implementation defined constructs, then the software cannot rely on any behaviour that is not exactly defined. Claims based on applications running on multiple platforms are only as good as the platforms are different. If the platforms are similar they are likely to behave in similar ways. If the platforms used are all very different (in compiler and OS, not just the cpu) then there is some chance that the application has had many of the non conforming constructs shaken out of it. But there is no proof that all of the nonconforming constructs have been removed.

Dealing with sales claims

It would be misleading to suggest that sales people are not telling the truth. They are telling the truth as they know it. Given that few purchasers ask serious questions about standards conformance the sales office is unlikely to have the answers to hand. Thus it is very important to give them as much information as possible to help them find the answer. The technical staff with knowledge of conformance issues are likely to be located in a hard to find office (bureaucratically speaking). Sales staff will need to be given exact questions to ask with an accompanying rationale for why these issues are important. They will have to explain to management, at various levels, why time needs to be spent on answering the question. So clarity and preciseness is important.

The purpose of this paper is to suggest pertinent questions that a procurer might ask a supplier and explain the reasoning behind these questions. Before continuing we first need to address the issue of what makes up an Open System. This discussion is not intended to provide yet another definition for this term. Rather it is here to ensure that we are all thinking along broadly similar paths and also to explain the framework behind the wording of standards that complicate the conformance issue.

What Open Systems?

So far we have been talking about the generic term Open Systems. What goes into an Open System? Well, POSIX is frequently associated with this term. Communications standards, under the umbrella term OSI, are also an important part of Open Systems. It is these standards that allow computers to communicate with each other. For the most part this paper will use POSIX in it's examples. In most cases OSI related standards could be substituted. POSIX itself is commonly thought of as a single standard. In fact it is the umbrella term for over 20 different standards (many still in draft form). Fortunately for its users common nomenclature is used throughout the POSIX documents, so it is possible to discuss them as if they were a single entity.

Depending on the application other standards are also likely to be applicable. There will be the language standard, in which the application is actually written, plus other domain specific standards. In the graphics arena we have GKS, PHIGS and CGI. There are the SQL and NDL standards for databases. The list of communications standards would be longer than this entire article. Windowing standards are the topic of some very hot debates.

POSIX was designed as a standard environment to enable the portability of applications software and to some extent people. This portability of applications software is achieved through the specification of a set of services that every POSIX conforming application can expect to exist on a conforming platform. As well as providing a defined set of services POSIX also specifies how they are to be used.

As already mentioned the requirements contained in standards are not all fixed, they can vary between implementations. In some standards the subsets of the services described may also be optional. POSIX.4 (real time extensions) is a good example of this. Virtually the entire standard is optional (each of the 15 components may or may not be available in a given, conforming, implementation).

To add further complexity, conformance to standards is not always a yes/no answer. In the case of the POSIX standard there are conformance requirements for both the platforms that provide services and for applications that use those services. POSIX defines four levels of applications conformance. These conformance levels range from strictly conforming to conforming with extensions.

  1. Strictly conforming
  2. Conforming
  3. <National Body> conforming
  4. Conforming with extensions
In the case of a strictly conforming application the behaviour of the software is fixed across all conforming platforms, a worthwhile property. Sometimes, however, applications programs need to make use of services defined in other standards. Any application program making use of such services can only be a conforming program.

Some of the types of non-conforming constructs that a program might use include:

Why don't applications conform to standards?

If standards only permitted conforming constructs in their requirements many portability problems would be solved. However, portability is only one of many competing claims on the design of a standard. Efficiency, easy of implementation and historical practice being some of the other topics that need to be considered.

Standards documents by themselves do not automatically ensure that platform vendors conform to the requirements that they contain. To ensure that platforms and tools do conform to standards, test suites are required. Work on validation suites (software to check conformance to standards is not simply called testing software) to check platforms for conformance to POSIX are well advanced. There are at least three such suites commercially available (for POSIX.1). Similarly validation suites exist to check C, Fortran, Ada, Cobol and Pascal compilers against their respective standards.

Having a standard and its associated validation suite is one thing, making sure that vendors make use of them is another. The difference in perception between a companies marketing and technical department on the subject of standard conformance is often wide. The only reliable guide to conformance is third party certification. There are currently seven accredited test laboratories offering POSIX.1 certification of platforms. But standards enforcement is not as direct as law enforcement. There are bodies whose job it is to monitor the enforcement of standards and laws. But the later body has the power to fine and throw violators is prison. Standards bodies do not have such strong arm powers and they have to rely on peer and customer pressure to keep vendors within and up to standards.

A more commercial reason than peer pressure, for following standards, is government procurement. The US government mandates that whenever possible items that conform to FIPS standards be given preference. The only accepted method of showing this conformance is via validation certificates. Because the government is such a large purchaser this conformance requirement has a trickle down effect to the smaller customers. In the UK the CCTA has a similar policy. However, the practical implementation of this policy is limited by the lack of a UK registrar of validated products (work is under way to remedy this problem).

POSIX offers the software developer the opportunity for a significant reduction in cost and effort when porting applications to different platforms. Given these benefits what stands in the way of creating POSIX conforming applications? There are two main reasons why applications fail to conform to the requirements of POSIX. The immediate problem is one of know how and old habits. Once these are overcome problems are caused by human oversight and error.

Because of the broad range of services offered it can take some time for developers to think POSIX. Old, Unix, programmer habits and know-how are easily transferred to a POSIX development environment. Programmers cannot be expected to be familiar with all the intricacies of POSIX and how it differs from what they are familiar with. Speaking Unix with a POSIX accent will not solve portability problems, particularly to proprietary platforms that support POSIX. It is necessary to speak POSIX as a native language and if using Unix perhaps with a Unix accent. Training can go someway towards ensuring a smoother transition to a POSIX only environment. But because it costs time and money to create, short cuts are often taken. Tools to do the job and training are the most frequent `savings'. Developers do not like to complain about this. The writing of portable software is rather like good driving, everybody thinks that they can do it. So in many cases the main problem that needs to be overcome is one of perception.

Extensions

Users want standard software that uses the latest technology. Such a requirement is a contradiction in terms. Standards take time to create and require broad consensus. Vendors are unlikely to share their technological advances with competitors. So initially, until it starts to succeed, new technology is only available from one company. Also given the long gestation period of standards new technology is unlikely to be formally endorsed for at least five years, usually ten.

New technology is created to improve on the past. It has lots of attractions. In some cases it may solve a previously unsolvable problem. So extensions have there use, but they should be recognised as such and their effect on standards conformance taken into account when deciding to provide a waver on a particular feature.

Multiple standards

By itself, the word standard is non-specific. There are company standards, industry standards, national standards and international standards. For those not immediately involved, the status of a particular standard can be a mystery. Through mutual agreements there tends to be only one international standard covering a given area. At the national level there are often competing standards (usually across national borders), but there is the intent to cooperate to move towards a single standard. Industry standards are usually controlled by a single company or small group of companies, cooperating to the extent that it will result in their own standard being used.

Questions to ask suppliers

Having outlined the issues behind creating standards conforming software we can now get down to the task of finding out exactly how products measure up. However, before asking any supplier questions about their products conformance it is very important that the questioner have already decided which standards are important and their level of importance in the purchase decision. It is no good asking for exact and detailed information if the questions themselves are inexact and wooly.

Conformance for purchases of hardware is a much easier matter to deal with that software. There is a US Government requirement that hardware conform to POSIX.1 (actually FIPS 152). To measure this conformance the National Institute of Standards and Technology (NIST) have set up a testing service and every quarter publish a list of conforming platforms. Given the availability of this service and the test suite that goes with it selecting the wheat from the chafe is easy, for hardware.

Applications software is a different story. At the moment there are no government requirements (with any teeth anyway) to purchase conforming application. Thus there are no accredited testing laboratories carrying out this task. The buyers of software currently have to fend for themselves. So what questions ought an intelligent buyer of applications ask of their vendor?

Asking a vendor if they conform to Open Systems is a waste of time. You will get the blanket answer "Yes". There is no formally agreed upon definition of the term Open Systems and everybody has their own definition for the phrase. So avoid it.

What is the correct first question? The best I know of is "What standards do you conform to?". A typical answer will waffle on about how useful standards are, how company X is involved with standards and of course the development groups are very interested in following standards. A clever salesman might even mention conformance to various standards, but forget to mention that they have nothing to do with the Open Systems, or the product at hand. Remember, the main reason for failing to conform to standards is caused by not measuring or testing the software. It is all very well a company being involved with standards development. In the first place the individuals involved are unlikely to be the same as those actually producing the products. Secondly unless standards documents, training and tools are made available to developers actually conformance is likely to be poor.

The purchasers best response is to follow up with a list of standards, asking if the product conforms to them. POSIX.1 is a good place to start, followed by POSIX.2, .3, ... POSIX.20 (I have never managed to get as far as twenty). Anybody claiming conformance to all of the POSIX standards (or drafts) is having you on. POSIX.10 relates to supercomputers, POSIX.3 relates to conformance testing and has nothing to do with applications software, POSIX.20 is the Ada binding to POSIX.1 and POSIX.9 is the Fortran binding (POSIX.19 is the Fortran-90 binding, how many languages is the application written in?). If you really want to throw numbers around you could get into the addendums to the standards (POSIX.1a, etc).

At the end of this initial conversation you will have a list of standards for which conformance is being claimed. The next question deals with these claims of conformance. Ask which of the four levels of conformance are being claimed. They might reply "The highest level of course". This is a bad answer. The highest level of conformance (strictly conforming applications) requires that "... only the facilities described in this part of ISO/IEC 9945 ..." be used. This implies that an application that strictly conforms can only use constructs from one standard. This requirement means that any strictly conforming applications is likely to be unusable (the user interface is not dealt with in POSIX, so how does the user access a strictly conforming application to make it do useful work?). Point this fact out to whoever is making such claims and ask for a more realistic statement of conformance.

You will now be in one of two positions. The vendor has either come clean and admitted that the application is not known to conform to any standards, "but lots of people are very happy with it", or they are insisting on some degree of conformance. In those cases where the vendor has given up any claims of conformance it is up to the purchaser to decide what to do next, that is a business decision. What of the vendor who claims some degree of conformance? The obvious response is to request proof. What tests have been carried out to show that the software does conform to standards? There are a number of tools available that will check applications for conformance to various standards. Have any of these been used?

Stages of conformance checking

As mentioned above many of the standards that go into the definition of Open Systems have only just been ratified or are still in draft form. There is always a time lag between the publication of a standard and methods of checking conformance to it becoming available. We are also living in an age where products are constantly being created and updated. There are few stable standards or products. For this reason we might expect conformance to various standards to be at in intermediate stage.

What stages of the conformance process might a product be at?

  1. The ideal stage is validated, without errors, by an accredited test laboratory. If this claim is being made ask to see the validation certificate. This will provide details of the version validated and when the testing was performed.

  2. Validations have to be booked in advance. A product might not yet have a validation certificate, but perhaps a formal validation has been booked. Ask to see the letter booking the validation with the test laboratory. Check the current status with the test laboratory.

  3. Prior to booking a validation a vendor will need to ensure that the product is going to pass. The way to do this is to obtain a copy of the test tools that will be used during the validation. Ask to see a copy of the purchase order for these test tools. A vendor that does not have these tools is in no position to make any conformance claims without first having the tools necessary to back these claims up. Of course if a vendor has the test tools but has not yet booked a validation ask to see a copy of the results produced by using the tools.

  4. In the worst case no test tools are available for a given standard, which is itself still in draft form. The best that can be done under these circumstances is to ask what draft of the standard is being implemented and what the plans are for following changes made in subsequent drafts.

The final question

The final question is likely to cause panic; "Can I see your conformance document?". One of the requirements contained in the POSIX standards for both platforms and applications is the requirement for the conforming and non-conforming aspects to be written down. I quote; "Such an application shall include a statement of conformance that documents all options and limit dependencies, and all other ISO or IEC standards used.". It is very unlikely that a vendor will have such a document. But it does leave them with something to do for your next meeting.

More information

If you are going to get involved with asking questions about Open Systems standards the first thing needed is a copy of the standards documents themselves. These can be obtained from BSI in this country, or the IEEE in the US. Because many of the POSIX standards are still in draft form they are not available to members of the public. One way of finding out what is going on is to attend standards meetings. In the UK the POSIX panel meets every three months.

The POSIX standards effort is so large that has spawned a standards document that explains what is contained in the other documents. This overview standard logically belong before the other standards, but rather than renumber the existing documents that decided to give it the number zero. The title of POSIX.0 is "Guide to the POSIX Open Systems Environment".

The DTI are also running various Open Systems awareness projects for industry. Contact the DTI at Queens Gate House in London for more information.

Conclusion

Vendors will continue to make spurious claims of conformance to Open Systems standards for as long as their customers are willing to accept them at face value. To find out the truth behind these claims users have to go through a process of discovery.

Before starting to ask vendors questions about standards conformance users should decide which standards they consider important. Having decided on a set of appropriate standards it is then possible to go onto ask a series of straight forward questions to find out the true state of vendors compliance with those standards.

At the end of the day standards compliance is only one of many issues that go into making the final decision. At this moment in time very few products conform to any standards. So be prepared to compromise. But if you are interested in products that conform let your vendor know that they will not get off so lightly next time.



© Copyright 1995. Knowledge Software Ltd. All rights reserved;
Home

Presented at the SunScope conference 1994