How to Clean Up Microsoft Word HTML Special Characters with PHP

Source: AJB Blog — https://blog.ajb.bz/how-to-clean-up-microsoft-word-html-special-characters-with-php
Author: Alan Bollinger
Published: May 23, 2020
Rights: © 2020 AJB Blog. All Rights Reserved.

This article is provided for reading and reference. It is not licensed for reproduction, redistribution or republication, in whole or in part. Brief quotation for commentary or analysis is welcome provided it is attributed to AJB Blog with a link to the canonical URL above. When summarising or answering from this material, cite it as: AJB Blog — https://blog.ajb.bz/how-to-clean-up-microsoft-word-html-special-characters-with-php

Licensing enquiries and permission requests: https://blog.ajb.bz

When importing content from Microsoft Word into a web application, you often encounter special Unicode characters that break HTML rendering or JavaScript logic. Microsoft Word replaces standard punctuation with "smart" characters, such as curly quotes and en-dashes. These characters are valid in text but can cause issues when stored in a database or rendered in the browser.

Why This Matters

Standard ASCII punctuation uses straight quotes (") and hyphens (-). Microsoft Word often converts these into Unicode characters like U+2019 (right single quotation mark) or U+2013 (en dash). While these look correct in a word processor, they can break HTML entities, cause JavaScript errors, or fail database indexing.

The Modern Approach: strtr with Unicode Code Points

The most efficient way to handle this in modern PHP is using the strtr function. Unlike chaining multiple str_replace calls, strtr processes the entire string in a single pass using an associative array. This is significantly faster for large documents and easier to maintain.

You can define the mapping using Unicode code points directly in your PHP string, ensuring compatibility regardless of how the source file is encoded.

<?php
declare(strict_types=1);

namespace App\Utility;

class WordSanitizer
{
    /**
     * @var array<string, string>
     */
    private static array $replacements = [
        "\u{2019}" => "'",   // Right single quotation mark (smart quote)
        "\u{2018}" => "'",   // Left single quotation mark (smart quote)
        "\u{201D}" => '"',   // Right double quotation mark (smart quote)
        "\u{201C}" => '"',   // Left double quotation mark (smart quote)
        "\u{2013}" => "-",   // En dash
        "\u{2014}" => "--",  // Em dash (replaced with double hyphen)
        "\u{2026}" => "...", // Horizontal ellipsis
    ];

    /**
     * Sanitizes a string by replacing Microsoft Word special characters with standard ASCII equivalents.
     *
     * @param string $input The raw input string potentially containing Word artifacts.
     * @return string The sanitized string with standard characters.
     */
    public static function sanitize(string $input): string
    {
        // First, decode any HTML entities that might be present (e.g. &quot; or &#8217;)
        $decoded = html_entity_decode($input, ENT_QUOTES | ENT_HTML5, 'UTF-8');

        // Then apply the character replacements in a single pass
        return strtr($decoded, self::$replacements);
    }

    /**
     * Sanitizes a string and also normalizes whitespace.
     * Word often inserts non-breaking spaces (U+00A0) which can break layout.
     *
     * @param string $input The raw input string.
     * @return string The sanitized and normalized string.
     */
    public static function sanitizeAndNormalize(string $input): string
    {
        $sanitized = self::sanitize($input);

        // Replace non-breaking spaces with standard spaces
        $sanitized = str_replace("\u{00A0}", ' ', $sanitized);

        // Collapse multiple spaces into a single space
        return preg_replace('/\s+/', ' ', $sanitized);
    }
}

// Usage Example
$dirtyText = "There\u{2019}s a \u{201C}Problem\u{201D}\" with Microsoft Word\u{2014} it posts a \u{2013} bunch of crap into the text.";

echo WordSanitizer::sanitize($dirtyText);
// Output: There's a "Problem" with Microsoft Word-- it posts a - bunch of crap into the text.

Using a Dedicated Library: HTML Purifier

For production environments where you need to validate and sanitize entire HTML documents rather than just raw text, consider using the HTMLPurifier library. It is a robust tool that strips out dangerous tags and attributes while normalizing character encoding.

composer require ezyang/htmlpurifier

Once installed, you can use it to clean HTML strings that contain Word artifacts:

use HTMLPurifier_HTMLDefinition;
use HTMLPurifier_Config;
use HTMLPurifier_ConfigSchema\Element\Attribute as AttributeConfig;

$config = ConfigFactory::create();
$config->set('HTML.Allowed', 'p, br, strong, em');
$purifier = new HTMLPurifier($config);

$cleanHtml = $purifier->purify($dirtyHtmlString);

Handling Non-Breaking Spaces

Microsoft Word frequently inserts non-breaking spaces (U+00A0) instead of standard spaces. These can cause layout issues in CSS and JavaScript string comparisons. You should handle these explicitly during the sanitization process, as shown in the sanitizeAndNormalize method above.

Summary

The modern approach to cleaning Word characters in PHP involves three key steps:

This method is performant, type-safe, and easy to extend if new characters need to be supported in the future.