In the world of web development, we've all deployed throwaway files for debugging—often with innocent names like test.php. But sometimes, these small conveniences can snowball into major security breaches.

One such case involved a test.php file containing a single, seemingly harmless line of code:

<?php phpinfo(); ?>

What’s the Harm in phpinfo()?
At first glance, phpinfo() is a useful tool. It provides detailed information about your PHP configuration, environment variables, installed modules, and more. But that’s also exactly why it’s dangerous.

When test.php was deployed and left publicly accessible, it revealed sensitive environment variables to anyone who visited the URL. Among those were AWS SES credentials—tokens that granted programmatic access to Amazon’s Simple Email Service.

The Consequence? Weaponized Infrastructure
An attacker discovered the file, harvested the credentials, and used them to integrate AWS SES into a mailer service. From there, they sent out a tidal wave of spam emails—all under the banner of the legitimate SES account. That not only led to a compromised sender reputation, but could have triggered account suspension, blacklisting, or even regulatory action depending on the volume and content of those messages.

Key Takeaways for Developers and Ops Teams:

• Never expose phpinfo() or debugging tools in production. Use secure environments and authentication barriers for diagnostics.
• Audit all public files before deployment. Even a single overlooked file can undo years of careful engineering.
• Use scoped IAM roles and do not store credentials in environment variables that are accessible by the web server.
• Set up alerts and rotate credentials regularly—a single leak shouldn’t mean total compromise.

🕵️ Other Files Scammers Commonly Hunt For

Bad actors often use automated scanners to sweep the internet for exposed files that can leak credentials, configurations, or system insights:

Some scanners even look for misconfigured CI/CD pipelines, logs, or temporary test scripts named debug.php, temp.php, or sandbox.php.

Debugging should never come at the expense of security. In this case, one line of PHP script created a gateway for exploitation. The next time you’re tempted to drop a test.php onto the server for a quick check—pause and think of the spam tsunami that might be lurking behind it.

2024 saw new peaks in usage for the sign language dictionary, www.signasl.org. With a whopping 34,412,937 searches conducted on the platform and 908,000 unique visitors. This was a 6.8% increase from 2023, which had 32,197,037 searches. This continued growth speaks to our commitment to providing comprehensive resources for those learning American Sign Language.

Top 50 Words Searched

Our top searched words for 2024 reflected both everyday conversation essentials and some intriguing trends:

• Pro
• Have
• Good
• Like
• What
• Want
• Love
• When
• How
• You're Welcome
• Do
• School
• Happy
• Thank You
• Where
• To
• And
• With
• Go
• Translate
• For
• Play
• Is
• Help
• Need
• Why
• Hello
• We
• How Are You
• Dog
• Favorite
• Work
• Please
• No
• I
• Nazi
• You
• Can
• All
• Brother
• Now
• Not

Upgraded IOS ASL App

We’re thrilled to announce the launch of our upgraded IOS ASL app, packed with user-friendly features to enhance your experience:

• New tabbed interface for easy navigation
• Access to Recent and Popular signs
• Option to hide video controls overlay
• Loop video playback for continuous learning
• Back and forward buttons in the video player for quick access to alternative signs
• Ability to continue audio playback from other apps while watching videos
• Show video attribution to credit content creators
• Save your favorite signs and view them offline
• Support for dark mode to reduce eye strain
• Improved startup behavior for better offline access

A special thanks to Mat Gadd for the incredible update! We couldn’t have done it without you.

As we move forward, we’re excited to continue providing improving the video dictionary. Stay tuned for more updates and happy signing!

Greetings, British Sign Language enthusiasts!

We're excited to share an incredible milestone in our journey to promote and support sign language learning. As we reflect on the year 2024, we’re thrilled to report some fascinating insights and achievements from www.signbsl.com.

Record-Breaking Searches: In 2024, our website experienced a significant surge in activity, with 536,288 unique visitors and a staggering 29,957,642 searches made. This is a 21.9% increase compared to the 24,587,983 searches in 2023. We are beyond grateful for the growing interest and engagement from our dedicated users worldwide.

Top 50 Searched Words: Our users’ curiosity and dedication to learning have highlighted some intriguing trends in search activity. Here are the top 50 words searched on our site in 2024:

1. how are you
2. dictionary
3. i love you
4. want
5. more
6. how
7. like
8. with
9. what
10. happy
11. when
12. thank you
13. no
14. good morning
15. why
16. do
17. toilet
18. and
19. have
20. love
21. tired
22. go
23. where
24. water
25. translator
26. my name is
27. holiday
28. favourite
29. help
30. can
31. school
32. need
33. today
34. you're welcome
35. hello
36. sorry
37. sign
38. beautiful
39. please
40. who
41. friend
42. dog
43. to
44. you
45. play
46. i
47. week
48. tomorrow
49. yes
50. work

These searches reflect the essential and heartfelt expressions that bind us together in communication. Whether it’s a simple greeting like “how are you” or expressing love with “i love you,” our users are learning to connect, share, and understand through the beautiful language of signs.

Upgraded IOS BSL App: 2024 also saw the release of our upgraded IOS BSL app, packed with user-friendly features to enhance your learning experience:

• New tabbed interface
• Recent and Popular signs
• Hide video controls overlay
• Loop video playback
• Add back and forward buttons to video player for quick access to alternative signs
• Don’t stop audio playback from other apps when playing videos
• Show video attribution
• Favourites
• View your favourites offline
• Dark mode support
• Improved app start-up behaviour for offline access

A big thank you to Mat Gadd for these fantastic updates that make our app more intuitive and enjoyable to use.

Looking Ahead: As we look to 2025, our goal is to continue growing and enhancing our platform. We have exciting plans to introduce new features, expand our dictionary, and provide more educational resources to support our users’ learning journeys.

Thank you for being a part of our community and for your support.

Here’s to another year of learning, growing, and connecting!

When managing a mailing list, one critical task is ensuring that the email addresses in your list are valid. Sending emails to invalid addresses not only clutters your system but also harms your sender reputation, which can affect your email deliverability. One effective way to validate email addresses is by checking their Mail Exchanger (MX) records.

What Are MX Records?

MX records are a type of Domain Name System (DNS) record that specifies the mail servers responsible for receiving email on behalf of a domain. If a domain has valid MX records, it means it is configured to receive emails. An email address without a valid MX record is considered invalid for sending emails.

Why Check MX Records?

1. Reduce Bounce Rates: Emails sent to addresses without valid MX records will bounce back. High bounce rates can negatively impact your sender reputation with email service providers.
2. Improve Deliverability: Cleaning up your list by removing emails without valid MX records helps ensure that your messages reach the intended recipients, improving overall deliverability rates.
3. Cost Efficiency: Sending emails to invalid addresses wastes resources. By identifying and removing these addresses, you can save on costs associated with email sending services.
4. Enhanced Analytics: A cleaner list provides more accurate metrics and insights. Knowing that your emails are reaching valid addresses helps you better measure the success of your email campaigns.

Bash Script

To help you can use the Bash script below to check if a domain list has valid MX records. If there is no MX record then any email sent here will be bounced. This can be used to remove email addresses that are not valid.

File domains.txt:

freebsd.org
redhat.com
yahoo.com
google.com

Then create mxlookup.sh

output='ns_output.txt'

# Clears previous output
> $output

# Seconds to wait between lookups:
loop_wait='1' # Is set to 1 second.

for domain in `cat $domain_list` # Start looping through domains
  do
    MX=$(dig MX $domain +short) #query MX records from domain list and store it as variable $MX
    #echo $MX >> $output;
    #echo $domain >> $output;
    arr=( $MX ) #creates array variable for the MX record answers
    echo ${domain} -- ${arr[1]} >> $output; #outputs only one record from above MX dig

    : '
    for ((i=1; i<${#arr[@]}; i+=2)); #since MX records have multiple answers, for loop goes through each answer
      do
        #echo $domain >> $output;
        echo ${arr[i]} >> $output; #outputs each A record from above MX dig
        #dig A +short "${arr[i]}" >> $output #queries A record for IP and writes answer
      done
    '

  done;

Run it ./mxlookup.sh which will produce a file ns_output.txt which will allow you to identify domains without a valid MX record. These can then be removed from your mailer list.

Python Script

Or if you prefer you can use this python script:

import dns.resolver

def check_mx(domain):
    try:
        records = dns.resolver.resolve(domain, 'MX')
        return True
    except (dns.resolver.NoAnswer, dns.resolver.NXDOMAIN):
        return False

domains = ["example.com", "invalid-domain.com"]
valid_domains = [domain for domain in domains if check_mx(domain)]
print(f"Valid domains: {valid_domains}")

By regularly checking the MX records of the domains in your mailing list, you can maintain a clean and efficient list, improve your email deliverability, and protect your sending reputation.

To send an email via a CloudFlare Worker you will need to call a email provider. AWS Simple Email Service (SES) is a robust transactional email provider that I already use in a number of other projects, and simplifies my billing. However the JavaScript SDK provided by AWS is very large and is not compatible with the CloudFlare Worker runtime. So we will want to roll our own simple API call to the SES.

Prerequisites and Setup

• Setup the domain identity in SES. This will require you to add 3 CNAME records to your domain name.
• You can then create a new IAM user with an IAM Inline Policy to restrict access to a particular email address.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ses:SendEmail"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "ses:FromAddress": "help@fiftypencedirectdebit.co.uk"
                }
            }
        }
    ]
}

• You will need to create access key for this new user, which will provide the credentials you will use in the code.
• For the contact sending we will use code from the AWS 4 Fetch project on GitHub - This project will handle the authentication required to use SES.

npm install aws4fetch

• Below is the code I am using. (Remember to change your SES region).

export async function sendEmail(toEmail, subjectLine, message) {
	const aws = new AwsClient({ accessKeyId: IAM_ACCESS_KEY, secretAccessKey: IAM_ACCESS_KEY_SECRET' });
	let resp = await aws.fetch('https://email.eu-west-1.amazonaws.com/v2/email/outbound-emails', {
		method: 'POST',
		headers: {
			'content-type': 'application/json',
		},
		body: JSON.stringify({
			Destination: 
			{
				ToAddresses: [ toEmail ],
				BccAddresses: [ 'me@daniel-mitchell.com' ],
			},
			FromEmailAddress: 'Fifty Pence Direct Debit <help@fiftypencedirectdebit.co.uk>',
			Content: {
				Simple: {
					Subject: {
						Data: subjectLine
					},
					Body: {
						Text: {
							Data: message.replace(/<br\s*[\/]?>/gi, "\n"),
						},
						Html: {
							Data: '<body><div align="center" style="font-family:Calibri, Arial, Helvetica, sans-serif;"><table width="600" cellpadding="0" cellspacing="0" border="0" style="font-family:Calibri, Arial, Helvetica, sans-serif"><tr style="background-color:white;"><td><table width="600" cellpadding="0"><tr><td><h1>Fifty Pence Direct Debit</h1><p>' + message +'</p></td></tr></table></td></tr></table></div></body>',
						}
					}
				},
			},
		}),
	});

	const respText = await resp.json();
	console.log(resp.status + " " + resp.statusText);
	console.log(respText);
	if (resp.status != 200 && resp.status != 201) {
		throw new Error('Error sending email: ' + resp.status + " " + resp.statusText + " " + respText);
	}
	return resp.status;
}

• Send the email via a function call:

ctx.waitUntil(sendEmail('Email To Address', 'Subject Line', 'Html Message here'));

The SES parameters that can be used in the API call can be found in the AWS documentation: https://docs.aws.amazon.com/ses/latest/APIReference-V2/API_SendEmail.html

Switching banks has never been more rewarding. With enticing cash bonuses, interest rates, and other perks, savvy consumers are capitalizing on these offers. However, many of these deals come with a requirement: you need to have active direct debits. But what if you could set up these direct debits without breaking the bank? Enter Fifty Pence Direct Debit, the service that makes it easy for you to manage your direct debits for just 50p per transaction.

What Is Fifty Pence Direct Debit?

Fifty Pence Direct Debit is a simple service to help those looking to take advantage of bank switch offers. Here's why:

1. Affordability: For only 50p per direct debit, it is the cheapest service to set up hassle-free direct debits for bank switching.

2. Speedy Setup: Need to switch banks quickly? Fifty Pence Direct Debit has you covered. Their simple signup ensures that you're up and running in no time. New direct debits are setup in 5 working days.

3. Flexible Usage: Whether you need one, two direct debits or more, Fifty Pence Direct Debit has got you covered. You can use the same service multiple times if necessary.

How It Works

1. Sign Up: Visit the 50p Direct Debit website and sign up. Provide your basic details, and you're ready to roll.

2. Pay Only 50p: Each direct debit setup costs just 50p. It's a small price to pay for the convenience and savings you'll enjoy.

Security and Peace of Mind

Fifty Pence Direct Debit takes security seriously. Here's why you can trust them:

• Payment Gateway: All payment processing is handled by the reputable payment gateway, Stripe. Your financial information is encrypted and secure.

• Cancellation Flexibility: Once your bank switch is complete, you have options. You can cancel the direct debit payment from your banking mobile app or leave it running for your next switch. It's entirely up to you.

Maximizing Bank Switch Offers

Now that you've set up your affordable direct debits with Fifty Pence Direct Debit, here are some recent of bank switch offers:

1. Santander: Get a FREE £175 cash bonus plus 1% cashback when you switch to Santander.
2. First Direct: Enjoy a FREE £175 bonus along with top-notch service.
3. TSB: Grab a FREE £100 bonus plus an additional £60 over 6 months.
4. Virgin Money: Benefit from 12.68% interest for the first 12 months.
5. Nationwide: Existing customers can earn a FREE £200 bonus.

Remember, these offers change over time, so always check the latest deals. With Fifty Pence Direct Debit, you're well on your way to maximizing your rewards without breaking the bank.

Conclusion

50p Direct Debit is your secret weapon for hassle-free direct debits. Switch banks, unlock bonuses, and keep more money in your pocket – all for just 50p!

When it comes to managing costs, especially ongoing ones, businesses and individuals with side projects alike seek efficient solutions. Email services are no exception. While free email providers can be enticing initially, they come with a serious drawback. Learn from my experience that see that resisting the draw of free email services and opting for paid transactional email services from the beginning is the smart choice.

The Free Service Dilemma:

• Free email services are convenient when setting up a project. They run smoothly for months or even years without any cost.
• However, there’s a catch: these free services eventually get discontinued. The abrupt announcement of it being discontinued can seriously disrupt your workflow, especially if you haven’t revisited the project in a while.

The Hidden Costs of Free Services:

When a free service is pulled, you’re left scrambling to:

• Retrieve the project details.
• Understand the deployment process.
• Perform necessary upgrades.
• Find, evaluate and transition to the new provider.
• Update DNS entries.
• Test and deploy.

These tasks consume valuable time and will completely negate any savings from using a free service that seems like such a good idea at the start.

The Solution: Always opt for paid transactional email services:

Here’s why:

• Minimal Cost: The expense of using a paid service is negligible compared to the time and stress saved when the service is inevitably pulled.

Personal Experience:

I have made the mistake of using a free email service twice:

• Once with MailChimp, which initially provided 5,000 free emails per month. Unfortunately, it was discontinued with just 30 days’ notice, prompting my move to AWS for my email needs.
• MailChannels in conjunction with CloudFlare Workers allowed unlimited free emails, but again they pulled the service abruptly. They even started immediately failing a small percentage of emails without any grace period, impacting my service. Again, I had to scramble to switch provider.

While I choose AWS because that is what I am already familiar with, any reputable email service provider will suffice. The important thing is to ensure that the service has long term support and will not leave you in the lurch by pulling the API or service.

In summary, investing in paid transactional email services ensures stability, peace of mind, and efficient workflows. The minimal cost is well worth the long-term benefits.

In 2023 32,197,037 searches were made on www.signasl.org a massive +22.5% increase from last year! (This is a 31% higher search volume compared to Sign BSL). Below I have provided a snapshot of the top 50 most searched words during the year.

Offensive language and racial slurs continue to see a significant increase in searches, with 7 of the top 10 most searched queries containing offensive terms, this is intriguing compared to the search data from the BSL version of the site where the top 50 searches do not contain any such offensive terms.

The big stand out search query for 2023 is searches for "abortion" which experienced a 758% increase from 2022, making it the 3rd most commonly searched word. This is not surprising given the political situation in the United States in 2023.

Aside from the offensive language, common and neutral terms such as "translate," "have," "good," "what," and "like" have also seen increases in searches, suggesting a diverse range of user interests and language learning activities. Polite expressions like "you're welcome," "thank you," and "please" have high search volumes, indicating a focus on communication etiquette.

Basic language components such as pronouns ("I," "you," "we"), conjunctions ("and," "or," "with"), and prepositions ("to," "for," "in") are frequently searched, reflecting a typical pattern of language learning.

Question words like "how," "when," "where," and "why" are commonly searched, suggesting users are interested in learning how to frame questions. Positive expressions like "love," "happy," and "peace" are searched frequently, indicating a desire for positive and uplifting conversations.

Other news updates

An update to the SignASL android app was released to fix video playback issues and support the latest versions of Android. (Special thanks to Gary Rennie for your support with this app.)

I would also like to thank Zachary Krueger a Sign Language Interpreter from Iowa who has for the last few months been reviewing and improving the quality of the ASL dictionary (removing incorrect signs and fixing the word definition associations for the videos).

For those interested in the technology stack that powers the ASL dictionary, I have upgraded the MySQL database from 5.7 to version 8. I have also upgraded the AWS Lambda functions from dot net core 3.1 to .net6. I have also replaced most YouTube embeds to use self hosted videos for improved playback experience.

Looking forward to what 2024 will bring for SignASL!

In 2023 24,587,983 searches were made on www.signbsl.com a +4.2% increase from last year indicating a growing interest in BSL which I expect to continue, especially with the plans to allow students to learn BSL as a GCSE subject. Below I have provided a snapshot of the top 50 most searched words.

Its pleasing to see that the common themes of the top searches include: Basic communication phrases, expressions of love and affection, inquiries about well-being, and seeking information. There is a strong interest in everyday communication. The top searches highlight the importance of BSL for everyday interactions and relationship building.

Other news updates

An update to the SignBSL android app was also released to fix video playback issues and support the latest versions of Android. (Special thanks to Gary Rennie for your support with this app.)

In preparation for the new GSCE in BSL I have added a new page to make it easy to find all the BSL GCSE vocabulary that students are expected to learn as part of their qualification.

For those who use Reddit, I have now setup daily auto posting of videos to the BSL subreddit. This posts the same videos as found on twitter.

For those interested in the technology stack that powers the BSL dictionary, I have upgraded the MySQL database from 5.7 to version 8. I have also upgraded the AWS Lambda functions from dot net core 3.1 to .net6. I have also replaced most YouTube embeds to use self hosted videos for improved playback experience.

Looking forward to what 2024 will bring for SignBSL!

As browsers start to phase out 3rd party cookies developers need to start using the partitioned cookie attribute (this is a new feature of Cookies Having Independent Partitioned State CHIPS). This will allow users to login when accessing your site in an iframe whiles maintaining privacy. While blocking 3rd party cookies is not yet set as the default, all the major browsers are planning to do this. Also to support users using Chrome in incognito mode you need to be ready to support this, otherwise you will be prevented from setting a cookie.

The partitioned attribute is supported in Chrome 114 and higher, it is also supported by Edge. How it works is well explained at https://developer.chrome.com/docs/privacy-sandbox/chips/ - essentially the cookie is partitioned according to the parent top level site. (See diagram below).

To configure this you need to set the cookie header:

Set-Cookie: __Host-name=value; Secure; Path=/; SameSite=None; Partitioned;

C# does not yet have an option to appended this Partitioned attribute, but you can just append the Partitioned property to the Path option. For example:

Response.Cookies.Append("X-Access-Token", accessToken, new CookieOptions()
{
    HttpOnly = true,
    Secure = true,
    SameSite = SameSiteMode.None,
    Path = "/; samesite=None; Partitioned"
});

This code also includes a fix to ensure the option samesite=None is outputted into the cookie (which is also required for 3rd party cookies).

One way to help reduce the chances of your emails being marked as spam is to understand email spam headers. Spam headers are the hidden messages that are attached to every email. They contain information about the sender, the recipient, and the content of the email.

Two of the most important spam headers are SPF and DKIM. SPF stands for Sender Policy Framework. It is a way for businesses to tell email servers which servers are authorized to send email on their behalf. DKIM stands for DomainKeys Identified Mail. It is a way for businesses to sign their email messages with a digital signature. This signature can be used to verify that the email message is actually from the sender it claims to be from.

What to check

1. SPF

SPF is a way for businesses to tell email servers which servers are authorized to send email on their behalf. This information is stored in a DNS record called the SPF record. The SPF record for a domain tells email servers which IP addresses are authorized to send email from that domain.

When an email server receives an email message, it will check the SPF record for the domain that the email message is from. If the IP address of the server that sent the email message is not listed in the SPF record, the email server will likely mark the email message as spam.

To check if this is setup correctly, in the message header you are looking for something that says:

Authentication-Results: spf=pass (sender IP is 159.135.225.169) smtp.mailfrom=daniel-mitchell.com;

If you can see spf=pass then this means the domain setup is correct. If not you will need to check the instructions provided by your email provider. They require a TXT DNS record that explains which IP addresses are allowed to send email on behalf of your domain.

2. DKIM

While not a requirement for email to be delivered to a users inbox it will increase its chances, and is well worth setting up correctly. The DKIM signature is created by the business that sends the email message. The signature is then included in the email message header. When an email server receives an email message, it will check the DKIM signature to verify that the email message is actually from the sender it claims to be from.

If the DKIM signature is valid, the email server will likely deliver the email message to the recipient's inbox. If the DKIM signature is invalid, the email server may mark the email message as spam or block it altogether.

To check if DKIM is setup correctly, in the message header you are looking for something that says:

dkim=pass (signature was verified)

If DKIM is not setup or the email is not signed, it will say:

dkim=none (message not signed)

To fix these issues you will need to check the instructions provided by your email provider. They require two CNAME DNS record that contain the DKIM signing key.

3. Other SPAM headers

This will vary according to the users email provider. They all have different rules and configuration. These headers are for users who use Microsoft365 for their email.

Email services use a number of rules and tools to determine the SPAM Confidence Level. This based upon the language and email style.

References
[1] https://learn.microsoft.com/en-us/exchange/antispam-and-antimalware/antispam-protection/antispam-stamps?view=exchserver-2019#the-phishing-confidence-level-stamp
[2] https://learn.microsoft.com/en-us/microsoft-365/security/office-365-security/anti-spam-bulk-complaint-level-bcl-about?view=o365-worldwide
[3] https://learn.microsoft.com/en-us/exchange/antispam-and-antimalware/antispam-protection/antispam-stamps?view=exchserver-2019#the-spam-confidence-level-stamp

Quick recap of this year's search data for SignBSL 23,595,029 searches were done. This is a tiny increase of +0.6% from last year. So almost unchanged the amount of searches. Below I have included the top 50 most searched words.

Quick recap of this year's search data for SignASL 26,274,632 searches were done. This is a small increase of +5.6% from last year. Below I have included the top 50 most searched words.

The top 5 words are offensive words. In fact there are 7 offensive words in the top 50, unlike users searching for BSL where none of the top 50 include such words. Of note from 2022 is the word 'Ukraine' enters into the top 50 with a 688% increase from last year. The word 'God' dropped from position 2 to position 42.

Many email organisers recommend a ruthless approach to email management. However, for me, email is an important record of all conversations and a record of what happened, it forms a journal of important details, which I rely upon when needed. Therefore deleting email is not an option. Yet, still finding emails that need to be answered or emails that represents a task I need to complete is important. So, being faced with a massive list that requires lots of scrolling is not optimal either.

The solution, is to create a customised view in Outlook. The only emails I want to see most of the time our emails that have not yet been read, or emails I have flagged because they represent some kind of task or action I need to take.

To do this take advantage of the Change View feature in Outlook. Click the little arrow three buttons in from the top left. And then choose Manage Views...

This brings up a new window which allows you to Manage Views.

We can then choose New. I give it a title of 90 days Unread or Flagged

Choose Filter

Select the tab SQL and then Edit these criteria directly and paste in the following statement.

(("urn:schemas:httpmail:read" = 0) OR ("http://schemas.microsoft.com/mapi/proptag/0x10900003" > 1)) AND ("urn:schemas:httpmail:datereceived" > today(-7776000))

This will then filter emails that are unread or have been flagged. And only if the emails are less than 90 days. (After all if you have not responded to them in 90 days they are no longer so important!)

Then choose this new view from the Change View icon.

Now you can enjoy the advantages of Inbox Zero of a small mailbox with only the most important messages displayed. Yet all those important emails that are a record of your life still intact. Enjoy!

Helping users find what they want fast is essential for a good user experience. Using a typeahead input box allows this. But to be effective it needs to be fast.

The fastest solution would be to have all the search entries available locally on the browser. But for this website there is currently 41,046 entries, taking 508kb of data. (gzip compression helps, but would still require 167kb of data to be sent over the wire). Sending this amount of data is not only slow, but also costly. With over 200,000 monthly users that would result in nearly 100GB of egress charges (33GB if compressed). Unlike website images this data can not be aggressively cached. For the typeahead search to be most useful this list needs to be regularly updated, not only for the latest search terms but also its order. Commonly searched or trending words need to be given priority in the list over less frequently searched words. To do this I update the search terms JSON file every 20 minutes.

So how can we make the typeahead fast but not burden the end user with a large JSON file? The trick is to bring that JSON file as close to the end users and then only transmit the entries they need to see as and when they request it. To do this I will use CloudFlare Workers, allowing me to leverage CloudFlare's global network of data centres. A Worker allows for a script size up to 1MB. This allows the whole dataset of all search terms to be stored. This script is then deployed to their edge locations around the world. The Worker can then take the search query requests and then filter the data, returning just a few rows to be displayed. CloudFlare provides up to 100,000 Worker requests per day, meaning that this performant, low latency typeahead can be deployed for free.

Setup

Select Workers from the CloudFlare menu on the left. Then choose Create a Service

Enter a name and choose HTTP Handler

Next under Triggers setup the path that this worker should respond on. In my case this was www.signasl.org/wordsearch.php* so that it could replace the existing server side method. This will mean a CloudFlare Worker nearest to the user will respond to the request rather than it having to travel to the single server in eu-west-1 region. I also set the Request limit failure mode to Fail open (proceed) this way if I exceed the free tier allowance then the typeahead will continue to work by falling back to its original (albeit slower) functionality.

Next step is to automate the updating of the Worker script including the large JSON array of search data. To do this you need to make a note of your CloudFlare AccountId, the name of the Worker Script and create credentials here that have permission to update this Worker script.

For me, I already have an existing C# AWS Lambda Function that I use to calculate recently searched data to make the typeahead ordering more helpful. This function already outputs the search options as a JSON file to S3. So I will add in an API call to this existing function to make an update to the Worker script.

The basic CURL command is:

curl -X PUT "https://api.cloudflare.com/client/v4/accounts/YOUR-ACCOUNT-ID/workers/scripts/typeahead-asl" \
     -H "Authorization: Bearer YOUR-TOKEN-HERE" \
     -H "Content-Type: application/javascript" \
     --data "SCRIPT DATA HERE"

My C# code in part looks like:

var requestContent = new StringContent(GetWorkerScript(jsonString), Encoding.UTF8, "application/javascript");
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", CloudFlareWorkerBearerToken);
client.PutAsync($"https://api.cloudflare.com/client/v4/accounts/{CloudFlareAccountId}/workers/scripts/{scriptName}", requestContent).Result;

The GetWorkerScript function produces the JavaScript output:

var globalData = ["WORD","LIST","ARRAY"];

addEventListener("fetch", event => {
    event.respondWith(handleRequest(event.request))
})

async function handleRequest(request)
{
    const url = new URL(request.url);
    if (!url.pathname.startsWith('/wordsearch.php')) return fetch(request);

    const { pathname, searchParams} = new URL(request.url)
    const query = (searchParams.get('query') || '').toLowerCase();
    
    var data = globalData;
    if (query != '')
    {
        data = data.filter(word => word.toLowerCase().includes(query));

        data = data.sort((a, b) => {
            // Give priority to exact matches
            if (a.toLowerCase() == query)
            {
                return -1;
            }
            if (b.toLowerCase() == query)
            {
                return 1;
            }

            const aPos = a.toLowerCase().indexOf(query);
            const bPos = b.toLowerCase().indexOf(query);

            if (aPos < 0)
            {
                if (bPos < 0)
                {
                    return 0;
                }

                return 1;
            }

            if (aPos < bPos)
            {
                return -1;
            }
            else if (aPos > bPos)
            {
                return 1;
            }
            else
            {
                return a - b;
                return 0; // neutral.
            }

            return 0;
        });

        if (data.length > 30)
        {
            data.length = 30; // Return just a small number of results for this specific search.
        }
    }
    else
    {
        // If no search result then return a large number of results to initially populate the typeahead search.
        data = data.slice(0, 4000);
    }

    response = new Response(JSON.stringify(data));
    response.headers.set('content-type', 'application/json;charset=UTF-8');
    return response;
}

The first line contains a Global Variable that lists all the search words ordered by how commonly searched for the word is. This Global Variable persists between executions giving a super fast in memory cache. Making subsequent requests lightning fast.

Only one Workers instance runs on each of the many global Cloudflare edge servers. Each Workers instance can consume up to 128MB of memory. Use global variables to persist data between requests on individual nodes as a super fast cache. Note: As the global variable is shared across requests, you must make sure you do not edit its content as that would then affect subsequent requests.

A basic filter at the beginning only respond to the expected requests. Although it should never respond to other requests as I have specified a specific request in the route setup.

Then it uses the array filter function followed by a custom sort which allows me to prioritise searches where matches appear nearer the beginning of the word. If someone searches for ma they are much more likely to be looking for man or mankind rather than looking for animal. If the position of the search term in the word is the same, then do not change the ordering, which will mean the pre-ordered list of words will return the most commonly searched for words.

This sorted list is then JSON.stringify and returned as JSON. Giving the following results when the user has entered ma:

["ma", "make", "made", "magic", "man", "management", "Mac", "Maker", "maths", "maybe", "make up", "March", "market", "Macintosh", "mainstream", "Makaton", "Malaysia", "manager", "math", "magnet", "magnetic flux", "main", "mango", "many", "masking", "May", "magazine", "mammal", "managing director", "map"]

Giving me responses between 30-40ms. With none of these requests hitting the origin server. Globally performance will be limited not by how close they are to the origin server, but solely based upon how far they are from a CloudFlare edge server.