Mailing out invitations - 2FA and API (Sendgrid)

More
4 months 1 week ago #207088 by davebostockgmail
I currently use SendGrid as the SMTP service to send out invitations and reminders from the site.

In a few months SendGrid is going to require either an API key or 2FA and not allow basic authentication to the site.

So my question is this ... is there any way to incorporate the API key into the mail settings in Limesurvey or does any one know of an alternative SMTP service.

We average around 250,000-300,000 invites / reminders per month which is why SendGrid was ideal.

Thanks in advance

Please Log in to join the conversation.

LimeSurvey Partners
More
4 months 1 week ago - 4 months 1 week ago #207098 by holch

We average around 250,000-300,000 invites / reminders per month which is why SendGrid was ideal.

Not an answer, but rather a question about your experience with Sendgrid. My experience with sendgrid and similar services is that they have a rather high rate of being marked as spam or even be straight refused from the mail server. e.g. our company email server does not accept any emails that come from them and with a lot of other tools we use for qualitative market research where the emails from the platform are sent via Sendgrid many of the invitation emails never arrive, which is why we also send the invitation via our company email address (which works fine because there are not many respondents in the qual research).

I answer at the LimeSurvey forum in my spare time, I'm not a LimeSurvey GmbH employee.
No support via private message.
Last edit: 4 months 1 week ago by holch.

Please Log in to join the conversation.

More
4 months 1 week ago #207109 by DenisChenu
Must create a plugin using github.com/sendgrid/sendgrid-php
and manual.limesurvey.org/BeforeEmail

Since they already done the wordpress plugin, maybe you can ask them if they can construct the LimeSurvey plugin ?

Assistance on LimeSurvey forum and LimeSurvey core development are on my free time.
I'm not a LimeSurvey GmbH member, professional service on demand , plugin development . I don't answer to private message.

Please Log in to join the conversation.

More
4 months 1 week ago #207116 by holch

Since they already done the wordpress plugin, maybe you can ask them if they can construct the LimeSurvey plugin ?


Hahahahahahaha...

I answer at the LimeSurvey forum in my spare time, I'm not a LimeSurvey GmbH employee.
No support via private message.

Please Log in to join the conversation.

More
4 months 1 week ago #207137 by davebostockgmail
Hi Holch

To get around the spam labels here are the steps we take.

1) Never use capitals in the subject lines
2) Always avoid spammy sounding subjects such as Win a Prize etc. they are always Help {Client} understand their business
3) We always create a valid email as the from address with the client in there This email address is being protected from spambots. You need JavaScript enabled to view it.

These help keep the self reporting down which SendGrid uses as part of the reputation score it geberates

Then within Sendgrid itself we have the account with the dedicated IP address that can be validated back to our mail server. This stops reverse lookup of email addresses being flagged as Spam

Finally we try as best as possible to clean the email addresses we get from the clients to ensure any This email address is being protected from spambots. You need JavaScript enabled to view it. and This email address is being protected from spambots. You need JavaScript enabled to view it. are removed as bounced emails also affect the SendGrid reputation.

Last month we sent 473,274 emails and our current reputation score is 92% within the SendGrid Dashboard.

If you are having issues with SendGrid not reaching the destination then a call to Support there will identify the reasons why. But first I would check your reputation score and then work through the outcomes on the dashboard as this can give insight into non delivery.

Hope this helps

Please Log in to join the conversation.

More
4 months 1 week ago #207138 by davebostockgmail
Denis,

This may have been possible before Twillio or whatever they are called bought them but not I doubt it very much.

Time to look for another provider I think.

Please Log in to join the conversation.

More
4 months 1 week ago #207153 by jelo

davebostockgmail wrote: Time to look for another provider I think.

In the next 12 months a lot of provider will switch to other authentification schemes. SMTP will be open to plain authentification the longest, but LimeSurvey will e.g. no longer connect via IMAP with the current authentification scheme.
Mostly big provider want to get 2FA into the game. Microsoft and Google will use OAuth 2.

SendGrid seems to offer different APIs. I wouldn't be surprised if they use OAuth2 too.

The meaning of the word "stable" for users
www.limesurvey.org/forum/development/117...ord-stable-for-users

Please Log in to join the conversation.

More
4 months 1 week ago - 4 months 1 week ago #207156 by DenisChenu

jelo wrote: SendGrid seems to offer different APIs. I wouldn't be surprised if they use OAuth2 too.

phpMailer already have Oauth.

This make a plugin more easy to construct :)

Maybe we need to move SMTP to a core plugin too ?

Assistance on LimeSurvey forum and LimeSurvey core development are on my free time.
I'm not a LimeSurvey GmbH member, professional service on demand , plugin development . I don't answer to private message.
Last edit: 4 months 1 week ago by DenisChenu. Reason: fix quote

Please Log in to join the conversation.

More
4 months 1 week ago #207160 by jelo

DenisChenu wrote: phpMailer already have Oauth.

phpMailer has a lot features. Question is what is used in LimeSurvey. I don't use LimeSurvey for mailings or Email-notifications. The development concept (core vs. core plugin vs. external/thirdparty plugin) of LimeSurvey is unknown to me. Without further knowledge I would prefer a core plugin for as many features as possible.

The meaning of the word "stable" for users
www.limesurvey.org/forum/development/117...ord-stable-for-users

Please Log in to join the conversation.

More
4 months 1 week ago #207163 by holch

If you are having issues with SendGrid not reaching the destination then a call to Support there will identify the reasons why. But first I would check your reputation score and then work through the outcomes on the dashboard as this can give insight into non delivery.

Thank you for the thorough answer. These are mainly thirdparty MROC platforms and we have no impact on how they configure their email settings. Of course this is a very small sample (and some respondents may just say they did not receive the email), and I am not keeping exact statistics but my feeling is that 25-50% of the participants do not receive those messages. They are usually not "spammy", but very tailored to the project, with information about what to expect.
But good to know that there is "hope", with the right configuration. My problem with those system emails is also that they usually have no bounce email system. So you send out invitations and sometimes the emails of the respondents have just been noted down wrong (qualitative recruitment). This has improved since most communication recruitment related run via Whatsapp these days, but believe it or not: People have difficulties to remember their email. Email for consumers here is dying. Soon those platforms will have to invite via Whatsapp directly... ;-)

I answer at the LimeSurvey forum in my spare time, I'm not a LimeSurvey GmbH employee.
No support via private message.

Please Log in to join the conversation.

More
4 months 1 week ago #207189 by blocka
I also use LS and SendGrid, with a similar number of email sends per month. I'd be interested in hearing what you plan to do to handle the changes and SendGrid.

And if a plugin can be developed for LS to allow continued use of SendGrid, I may be able to assist in funding its development.

Please Log in to join the conversation.

More
4 months 1 day ago #207499 by ViliusS
Just to confirm that two biggest email providers Microsoft and Google is moving to deprecate Basic Authentication for IMAP/POP and later SMTP AUTH.

For Gmail accounts the only path forward with IMAP/POP will be to use OAuth gsuiteupdates.googleblog.com/2020/03/les...n-off-suspended.html The switch off was planned to happen this year but was postponed to to COVID-19, however will probably resume next year. SMTP will work for sometime, though.

For Office 365 accounts Basic Authentication for IMAP/POP will be disabled as early as the first part of 2021. techcommunity.microsoft.com/t5/exchange-...-update/ba-p/1530163 SMTP will also work for sometime, but don't consider it to be forever.

I definitely think that LimeSurvey needs to implement OAuth in the core product. Times have changed and soon the only way to send email using just username and password will be to have your own server, which is not an option for large quantities and doesn't provide real delivery reporting capabilities.

Please Log in to join the conversation.

More
4 months 22 hours ago #207501 by DenisChenu

ViliusS wrote: which is not an option for large quantities and doesn't provide real delivery reporting capabilities.

gmail or microsoft are not the 1st choose for mass mailing.

There are a lot of existing services for mass mailig. Currently they have SMTP.

Else : OAuth can be in plugin, like SMTP in fact … and LimeSurvey ouit of the box can offer only mail.

Like Wordpress for example …

Assistance on LimeSurvey forum and LimeSurvey core development are on my free time.
I'm not a LimeSurvey GmbH member, professional service on demand , plugin development . I don't answer to private message.

Please Log in to join the conversation.

More
4 months 21 hours ago - 4 months 21 hours ago #207505 by ViliusS
I'm not talking only about SMTP or these two particular services here. I'm talking about email services *trend* in general. Bounce functionality, which requires IMAP or POP mailbox access will be affected too, which probably can't be solved by plugins.
Last edit: 4 months 21 hours ago by ViliusS.
The following user(s) said Thank You: DenisChenu

Please Log in to join the conversation.

More
4 months 21 hours ago #207506 by DenisChenu
Right, sorry …

And currently : no way to extend … maybe need to move whole bounce system to a plugin ? :P :whistle: ;)

Assistance on LimeSurvey forum and LimeSurvey core development are on my free time.
I'm not a LimeSurvey GmbH member, professional service on demand , plugin development . I don't answer to private message.

Please Log in to join the conversation.

Start now!

Just create your account and start using Limesurvey today.

Register now