Keywords

1 Introduction

1.1 Background

Composite materials are used in various industrial applications and for light high-performance structures, for example in the automotive industry. Main reasons are the advanced specific mechanical properties compared to metals. For example, Kim et al. have designed a automotive lower arm which is 50% lighter and much stiffer than the conventional steel part [9]. However, the production of composite materials is still rather expensive. A significant part of the production costs are associated with quality inspection, as this is still conducted as manual labour [13]. An automated inspection system therefore carries a big potential to drastically reduce the manufacturing costs of composite parts.

In this work an inspection system for the Dry Fibre Tape Placement process is proposed. It is based on a laser line triangulation sensor which captures the surface of the fibre layup in-line and outputs a point cloud. A challenge of this application is to process the data in order to detect and classify defects in the fabric. In the literature, multiple different approaches from the domain of machine vision are considered. Multiple authors summarised and compared the methods, for example in the field of industrial fabric production [6, 10, 14], the medical domain [11, 15], or the Automated Fibre Placement process [16]. However, as Mahajan et al. stated, there is not one best approach for all applications. Rather, for each use-case a suitable method must be selected, considering the specific process characteristics [14]. The inspection of dry fibres bear the challenge that the surface in not homogeneous. A layup without resin contains small out-of-plane waviness and some fibres sticking out of the tape. Furthermore, some dry fibre tapes already contain slits to enhance the resin flow during the infusion step. These kinds of irregularities are tolerable. On the other hand, gaps and overlaps must be detected as they influence the local fibre volume fraction and consequently the final mechanical properties [2]. So far, there is no report about an implementation of a method, that reliably differentiates defects (such as gaps and overlaps) from acceptable irregularities in dry fibre layups. The aim of this paper is to answer the following research question: Which evaluation method can reliably detect gaps and overlaps in a dry fibre tape layup, which contains strong irregularities (such as slits, waviness etc.) and accurately classify them as defective?

1.2 Dry Fibre Tape Placement

Production processes for composite materials can be divided into two main categories: pre-preg (short for: pre-impregnated) methods, where the fibres have been mixed with the resin before they have been brought into the final shape, and dry fibre processes, where the resin is added at the end of the process in an infusion step. While some pre-preg processes, such as Automated Fibre Placement, are established in the industry, they also require an expensive material handling under freezing temperatures [22]. Dry Fibre Tape Placement on the other hand promises to be an inexpensive production approach with little scrap material and low cycle times [5]. In this work, the FILL Multilayer (see Fig. 1 left) is used to deposit straight strips of bindered dry fibre tape onto a vacuum table with 360 degrees of freedom. It uses multiple laying heads simultaneously to lay the tapes. Each laying head performs 1-dimensional movements on linear rails. This process creates a near-net-shape layer of carbon fibre. After each layer the table can turn the layup and another layer with a different fibre orientation can be added. The tapes of the different layers are held together by thermoplastic binder that is activated by local ultrasonic welding.

Fig. 1.
figure 1

Left: Multilayer with multiple laying heads running on rails. Right: Setup of the laser line triangulation sensor on a 6-axis robot arm.

Full size image

1.3 Defects and Irregularities

At the stage of the 2D layup, many different irregularities can be observed. On the grey scale image of a defect-free specimen (compare to Fig. 2 left), an overall gradient in height from left to right is clearly visible. This is caused by the stiffness of the tape which was previously wound on a spool and tends to spring back out of plane after it was laid on the table. The oval shaped darker spot towards the right edge of the image is the dent, which the welding probe punches. To melt the binder, the tip of the ultrasonic device is pushed into the layup. Left of this dent a slit can be seen which was introduced by the tape’s manufacturer in a regular pattern to enhance the resin flow during the infusion step. All over the image slight lines running from left to right indicate fibres that slightly stick out compared to their neighbours. All these irregularities are inherent in the process and acceptable towards the quality of the final part.

In between neighbouring tapes however, gaps and overlaps can be observed in this process. These can occur if the tape is locally too narrow (which leads to gaps) or too wide (which leads to overlaps). Furthermore, the tape usually has some lateral clearance on both sides in the laying head to prevent it from getting stuck. This might also lead to slight deviations in the positioning of the tape, resulting in gaps or overlaps. Both of these defects are detrimental towards the final part’s mechanical properties, because they change the local fibre volume fraction [2].

Further defects that have been reported to occur in dry fibre tape placement are fold-ups, waviness, and early/late cuts [25]. These however are not considered in this work as they have not been observed regularly in the FILL Multilayer process or can easily be compensated by trimming.

Fig. 2.
figure 2

Grey scale image of three specimen: defect-free (left), with a gap (middle), with an overlap (right). The grey scale represents the height value.

Full size image

1.4 Laser Line Triangulation

A laser line triangulation sensor projects a laser line vertically onto the specimen’s surface. In an angle to this beam, a camera records the projection. Therefore, height differences in the specimen can be identified as a lateral shift on the taken image. The sensor’s software calculates a height profile of this image. As the sensor is linearly moved over an area of interest, it captures multiple profiles. These individual data sets can be reassembled to create a 3 dimensional representation of the specimen’s surface.

In research laser line triangulation sensors are commonly used for the quality inspection of composite parts. Unlike other inspection technologies, such as CT or optical systems, it can perform an in-situ inspection without adding an additional process step. The method has been verified with different carbon fibre materials, such as dry non-crimp fabrics [21], dry tows [26] and pre-preg tapes [1, 16, 20].

1.5 In-Line Implementation

In order to implement an in-line quality inspection of the layup during the production, a laser line triangulation sensor will be attached to a laying head and travels with it on the linear rail. This enables in-situ data collection during tape placement and avoids disruptions in the production process. Another advantage is, that possible defects are always oriented in the same direction relative to the sensor. In Fig. 2 the scanning direction will always be horizontal.

2 Literature and Methods

2.1 Evaluation Methods in the Literature

A commonly used inspection method for fibre layups is the optical inspection, where a camera takes a high resolution image of the fibre layup. This image is then evaluated. Focke et al. proposed to define a gap as an area where the fibre orientation is near to perpendicular to the dominating fibre orientation in the image [4]. The fibre’s direction can be determined by for example a fast-Fourier-transformation [18] or direct tracking [17]. Both methods require an image quality which is sufficient to identify the individual fibres. For this, the camera can only take a small segment of the layup at once to provide a sufficiently high resolution [7]. Furthermore, the camera must stand completely still and vibration free or the image might be blurred. Hence, an optical system is not suitable for an in-line inspection on a moving laying head.

Laser line triangulation sensors, on the other hand, have been proven to provide good data while they are moved over the specimen. Some commercially available laser line triangulation systems take the positions of the measured points and apply an one- or two-sided threshold to the absolute z-value [8, 23]. Some other systems can compare the measured points to CAD data and threshold the distance between the two data sets [3]. However, these methods proved to be inadequate to detect small defects in carbon fibre layups. Therefore, researchers have adopted algorithms from other domains. Most commonly, the 3 dimensional point cloud is converted into a 2 dimensional grey-scale image, where the z-coordinate is represented as brightness of the individual pixels. This bears the advantage that image processing algorithms from different domains can be applied.

Hanbay et al. investigated various image processing methods for fabric inspection and classified them into five categories: “structural”, “statistical”, “spectral”, “model-based”, “learning” and “hybrid models” [6].

Structural approaches require a definition of an arrangement of different primitive texture elements. This can be lines or dots on the image which should appear in a regular pattern. The algorithm checks whether these patterns are broken. Hanbay et al. state, their “reliability [...] is low. Structural approaches are only reliable in segmenting fabric defects from texture whose pattern is very regular” [6, p. 11963]. Laser scan images of dry fibre tapes, unlike woven materials for example, usually don’t have a regular pattern, as can be seen in Fig. 2. This is why this group of algorithms is not applicable to dry fibre tape laying. Statistical approaches are based on statistical evaluation of the image data to extract features. The absolute threshold method is one of them. Furthermore, many edge-detection methods are part of this group. To mention are the morphological edge detection [16, 24], gradient thresholding methods using the Sobel, Laplace and Scharr algorithms [12, 16], cell-wise thresholding [16] and adaptive thresholding [10, 14, 16]. Another statistical approach is the image projection where the brightness values of the image are reduced to a graph. This can then be evaluated with basic analytical functions. Hanbay et al. showed that the position of an elongated defect in woven material can be determined in one dimension of the grid [6]. Lastly, a method using co-occurrence matrices must be mentioned in this group. This approach detects recurring patterns in the image. Spectral approaches intend to detect regular or homogeneous patterns in the image. The earlier mentioned fast-Fourier-transformation is one of them. Other methods in this group are wavelet- or Gabor-transformations. The resolution of the laser line triangulation data is not sufficient to identify individual fibres. For this reason, spectral approaches cannot detect the fibre directions or defects. Model-based approaches require a model, which describes and constructs the expected data. This model is then compared to the measured data to detect unwanted areas [6]. Römer states that these “control systems are very complex to develop because many different effects have to be considered” [19]. Especially in the dry fibre tape placement, where the width deviations of the tape depend on the manufacturer’s process, it is extremely difficult to create a model that predicts gaps or overlaps. Irregularities such as slits and individual fibres sticking out are close to randomly distributed. Therefore, creating a model is infeasible. Learning approaches are methods, where networks are taught to detect defects. They require a great amount of data. Creating this labelled training data experimentally is very expensive and time-consuming. For this reason, Zambal et al. chose to generate artificial data using a probabilistic model. However, they found that the network performed worse when it was trained on artificial data [26]. The approach, which this project follows is to find a suitable statistical classification algorithm which can then label sets of training data automatically. Then, at a later stage, learning approaches will be considered and included in the research.

Meister et al. compared different algorithms for their use-case of the automatic fibre placement of pre-preg fibres. They assessed 29 commonly used evaluation methods to find a suitable algorithm. 5 options were investigated in-depth. Finally, they concluded that adaptive thresholding and cell wise standard deviation thresholding suit the requirements best [16].

2.2 Research Method for the Algorithm Assessment

In this research, the projection method was chosen, as it was proven to detect long defects [6]. Gaps and overlaps span along the whole width of the specimen. Therefore, it is assumed that this method can be suitable. However, in this study it is compared to the most suitable methods from Meister et al. paper. For this, 63 images of scanned dry fibre tape layups were fed into the algorithms. These images contained either no defects (21 samples), a gap (21 samples) or an overlap (21 samples). It is assessed whether the algorithms detect the flaws correctly. The measures were the true positive rate (defect found correctly), and the false positive rate (defect found on a defect-free sample). Furthermore, it was investigated whether the size of the defect can be assessed.

2.3 Specimen, Hardware and Preprocessing

The specimen were produced by the FILL multilayer. The material used for the specimen is a dry fibre, uni-directional carbon fibre tape by MTorres. Its areal weight is 450 g/m2 and its thickness is on average 0.52 mm. For the defect free samples, an area in the middle of one tape was taken. This way it was made sure, that it did not contain any gaps or overlaps but only irregularities, which are acceptable. For the gaps and overlaps two tapes where placed next to each other with a defect size of 1, 3 or 5 mm (7 samples each). The defect was placed roughly in the middle of the scan.

To acquire the data a preliminary hardware set-up is used, which imitates the movements of the laying heads. A commercial laser line triangulation sensor (Keyence LJ-X8200) scanned the specimen. This device works at a distance of 245 mm from the specimen with a 72 mm field of view. Each profile provides 3200 points, which is equivalent to a resolution of 44 dots/mm in x direction. For the measurement, a 6-axis robot arm performed a linear, homogeneous movement at a speed of 10 mm/s. At a measurement frequency of 1 kHz this results in a resolution of 100 dots/mm in y direction. During the measurement the sensor head provides the data to an evaluation unit (Keyence LJ-X8000A). The set-up is illustrated in Fig. 1 right.

On the software side, the following process is used to pre-process the acquired test data: As a first step, the data is collected using Keyence’ software LJ-X Navigator in order to receive a point cloud for further evaluation. This point cloud is cropped in x/y direction to a size of 50 \(\times \) 50 mm2. Furthermore, all outliers in z-direction which might be caused by reflections are removed. In a next step, the points are projected on a 2000 \(\times \) 2000 pixel grid. If multiple points fall on the area of a pixel, the average z-value is taken. If, on the other hand no point falls into the pixel, the average of the surrounding pixels is calculated. To convert the point cloud into a grey scale image, the z- value is mapped on a 8-bit grey scale. The contrast equalisation is performed with the CLAHE algorithm to compensate for uneven intensity distributions. Figure 2 shows the pre-processed scans.

3 Results

3.1 Image Projection

The image projection algorithm was programmed to calculate the average grey scale value of every row of the image \( R_\textrm{r}(i) = \sum _{j=0}^{n}(I(i,j))\). This graph can be seen in Fig. 3. If this graph showed an average slope of more than 2 Intensity steps per pixel over at least 14 pixels (\(\frac{\sum _{i=k-6}^{k+7}R'_\textrm{r}(i)}{14} >2 \)), this area was marked as a suspicious area.Footnote 1 If there is a pair of suspicious areas, which have opposite-signed slopes, they are identified as a gap. All single suspicious areas are defined as overlap. One example result for a specimen with a gap can be seen in Fig. 3. If an image contains either a gap or an overlap, it is classified as defective.

Fig. 3.
figure 3

Top: Successful defect detection using Image Projection. Left: grey scale image, middle: average grey scale values for each row, right: detected gap. Bottom: False detection of an overlap

Full size image

Out of the 63 samples, all defective samples were classified as such. The true positive rate is 100%. However, on 7 samples an additional defect was detected, where none was expected. 3 of these were on samples that had a defect at another position. The remaining 4 were on defect free sample. Therefore, the false positive rate is 19%. The confusion matrix can be seen in Table 1. All of the 7 false reports can be explained with a slit, which was extraordinary deep at this position.

Table 1. Confusion matrix for defect classification using image projection
Full size table

In addition to this classification, the algorithm also calculates the width of a gap once detected. The standard deviation between the produced gap width and the measured gap width is 0.12 mm. This accuracy includes some manufacturing tolerances, which are in the same range. This value has not been cross-checked with another measurement system to determine the true size of the gap. However, this feature to measure the gap width is useful if the requirements are changed. If, for example, overlaps and gaps smaller 2 mm are allowed, the true positive rate is 100% and the false positive rate 0%. This means that all previously falsely detected defects are smaller than 2 mm.

3.2 Adaptive and Cell-Wise Thresholding

Adaptive Thresholding is an approach, where no global threshold value is set. Instead the threshold for each position on the picture is determined by the neighbouring pixels. A mean (or Gaussian-weighted mean) of all pixels in a predefined radius is calculated and the threshold is defined based on the result. Cell-wise thresholding works similarly, with the difference that calculation is faster due to the predefined grid. Furthermore, the standard deviation of the pixel values is incorporated, which makes the method more accurate in specific use-cases [16].

Fig. 4.
figure 4

Processed grey scale image (left) with different evaluation methods: Adaptive thresholding, cell-wise thresholding, gradient analysis (left to right)

Full size image

Both methods were implemented and applied to the test data. It was found that all defects can be detected if the thresholds are selected sufficiently low. However, many small irregularities lead to false classifications. The algorithm marks all pixels that deviate from their neighbours, without differentiating whether this pixel is part of a major defect or a local deviation, such as a fibre sticking out of plane. This can be seen in Fig. 4 where, even in defect-free areas many pixels exceed the threshold and are thus marked as defects. In fact, all images of the data set where marked defective.

In order to lower the false positive rate, the threshold was increased and the image filtered and smoothed to reduce the severity of irregularities. However, no set of parameters was found that leads to a true positive rate of 100% and a false positive rate of <100%. In other words, either the algorithm misses multiple defects, or it is marks every image as defective. This paradox can be explained by analysing an image that shows the local gradients of the data set (see Fig. 4 right): Some irregularities have a higher gradient than the defects. Therefore, any set of parameters that detects the defect, also marks these irregularities as defective.

4 Discussion

In the pre-processed images, the inhomogeneity of the surface of dry tape layups can clearly be seen. Depending on the requirements, these irregularities (out-of-plane spring back, waviness and slits) might not be considered defects even if their local magnitude is similar to critical defects, such as gaps and overlaps. However, they are the reason why the algorithms that work best on pre-preg tapes (compare to [16]) fail on dry fibres. Above all, slits, which were introduced by the tape manufacturer, make a correct classification difficult if algorithms are used that threshold the slope of the edges or the magnitude of local pixels.

Spectral approaches have been proven to be applicable on high-resolution optical inspection systems [4, 18]. They are based on the difference of fibre orientation between stacked layers. However, a sufficiently sharp optical image cannot be taken while the laying head is moving. Furthermore, the resolution of laser line triangulation sensors is too low to identify individual fibres. This is why the commonly used fast-Fourier transformation cannot be applied.

The proposed method of image projection makes use of the fact, that the relevant gaps span the whole image from left to right and are always parallel to the image’s edges. Using these properties a classification with a true positive rate of 100% and a false positive rate of 19% can be reached. This accuracy exceeds the results from the literature in both, the false negative rate and the false positive rate (number based) [16]. However, these results also mean, that every fifth dataset is falsely classified defective. Therefore, a further inspection step is needed to reassess the results - this could either be a manual or an optical investigation. However, the proposed method is still useful, as the localisation of possibly defective areas accelerates the second inspection step drastically. It can be concluded, that this approach is a step towards a faster and cheaper inspection of dry fibre layups but does not yet replace other inspection steps.

The application of the image projection method is not limited to the FILL multilayer, but can be considered for all use-cases with dry fibres, where the laying heads run on straight paths. The irregularities, which prevent other evaluation methods from a proper classification, occur in most dry fibre tape processes.

5 Conclusion and Outlook

The aim of this research was to identify an evaluation algorithm that could detect gaps and overlaps in dry fibre tape layups. These contain irregularities such as an overall slope, waviness, slits or single fibres sticking out of plane. These irregularities make the application of adaptive and cell-wise thresholding methods impossible, even though they have been proven to perform well on pre-preg tape layups. Instead the image projection method is proposed which makes use of the uniform shape and orientation of the defects. When applied to a test set of specimen that contain gaps, overlaps and defect-free layups, the true positive rate is found to be 100%, while the false positive rate is 19%. Hence, this method cannot yet replace an additional process step for the inspection, but make it faster by preselecting possibly defective areas. This can be beneficial towards the process time and cost.

In future works, the parameter will be optimised further to improve the false positive rate. This algorithm will then be used to create labelled training data for a machine learning approach. Furthermore, the hardware system will be transferred from the test set-up (mounted on a robot) and be implemented in a Dry Fibre Tape Placement machine. The goal is to scan and evaluate the layup in real time to provide an in-line inspection system for this manufacturing process.